2025-02-17 02:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+802864414 ms)
2025-02-17 02:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 02:56:25.400206+00:00', 'runtime': '802860.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.501', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.299', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+802864415 ms)
2025-02-17 02:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 02:55:25.400197+00:00', 'runtime': '802800.001', 'adc_vb_f': '12.299', 'adc_va_f': '1.501', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.302', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+802864415 ms)
2025-02-17 02:56:35.631 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+802873451 ms)
2025-02-17 02:56:35.632 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+802873451 ms)
2025-02-17 02:56:35.633 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 02:56:25.400086+00:00'} (T+802873453 ms)
2025-02-17 02:57:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+802924404 ms)
2025-02-17 02:57:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+802924405 ms)
2025-02-17 02:57:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 02:57:25.400221+00:00', 'runtime': '802920.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.501', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+802924406 ms)
2025-02-17 02:57:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 02:56:25.400206+00:00', 'runtime': '802860.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.501', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.299', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+802924406 ms)
2025-02-17 02:57:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+802933445 ms)
2025-02-17 02:57:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+802933445 ms)
2025-02-17 02:57:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 02:57:25.400216+00:00'} (T+802933446 ms)
2025-02-17 02:58:13.353 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+802971172 ms)
2025-02-17 02:58:13.353 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+802971172 ms)
2025-02-17 02:58:13.354 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48228)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+802971173 ms)
2025-02-17 02:58:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+802984410 ms)
2025-02-17 02:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+802984410 ms)
2025-02-17 02:58:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 02:58:25.402790+00:00', 'runtime': '802980.003', 'adc_vb_f': '12.299', 'adc_va_f': '1.501', 'adc_vl_f': '12.283', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+802984411 ms)
2025-02-17 02:58:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 02:57:25.400221+00:00', 'runtime': '802920.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.501', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+802984412 ms)
2025-02-17 02:58:35.913 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+802993732 ms)
2025-02-17 02:58:35.913 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+802993733 ms)
2025-02-17 02:58:35.914 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 02:58:25.400200+00:00'} (T+802993733 ms)
2025-02-17 02:59:21.515 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803039335 ms)
2025-02-17 02:59:21.516 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803039335 ms)
2025-02-17 02:59:21.516 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48302)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803039335 ms)
2025-02-17 02:59:26.820 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803044640 ms)
2025-02-17 02:59:26.821 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803044640 ms)
2025-02-17 02:59:26.822 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 02:59:25.402817+00:00', 'runtime': '803040.003', 'adc_vb_f': '12.296', 'adc_va_f': '1.501', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803044641 ms)
2025-02-17 02:59:26.822 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 02:58:25.402790+00:00', 'runtime': '802980.003', 'adc_vb_f': '12.299', 'adc_va_f': '1.501', 'adc_vl_f': '12.283', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803044642 ms)
2025-02-17 02:59:31.600 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803049419 ms)
2025-02-17 02:59:31.600 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803049419 ms)
2025-02-17 02:59:31.600 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48314)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803049420 ms)
2025-02-17 02:59:35.643 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803053462 ms)
2025-02-17 02:59:35.644 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803053463 ms)
2025-02-17 02:59:35.645 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 02:59:25.400217+00:00'} (T+803053464 ms)
2025-02-17 03:00:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803104417 ms)
2025-02-17 03:00:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803104418 ms)
2025-02-17 03:00:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:00:25.402639+00:00', 'runtime': '803100.003', 'adc_vb_f': '12.296', 'adc_va_f': '1.505', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803104420 ms)
2025-02-17 03:00:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 02:59:25.402817+00:00', 'runtime': '803040.003', 'adc_vb_f': '12.296', 'adc_va_f': '1.501', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803104420 ms)
2025-02-17 03:00:35.715 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803113534 ms)
2025-02-17 03:00:35.715 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803113534 ms)
2025-02-17 03:00:35.716 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:00:25.400028+00:00'} (T+803113535 ms)
2025-02-17 03:01:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803164421 ms)
2025-02-17 03:01:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803164422 ms)
2025-02-17 03:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:01:25.402725+00:00', 'runtime': '803160.003', 'adc_vb_f': '12.296', 'adc_va_f': '1.511', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803164423 ms)
2025-02-17 03:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:00:25.402639+00:00', 'runtime': '803100.003', 'adc_vb_f': '12.296', 'adc_va_f': '1.505', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803164424 ms)
2025-02-17 03:01:35.801 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803173621 ms)
2025-02-17 03:01:35.802 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803173621 ms)
2025-02-17 03:01:35.803 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:01:25.400095+00:00'} (T+803173622 ms)
2025-02-17 03:02:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803224410 ms)
2025-02-17 03:02:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803224410 ms)
2025-02-17 03:02:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:02:25.401134+00:00', 'runtime': '803220.002', 'adc_vb_f': '12.292', 'adc_va_f': '1.517', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803224411 ms)
2025-02-17 03:02:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:01:25.402725+00:00', 'runtime': '803160.003', 'adc_vb_f': '12.296', 'adc_va_f': '1.511', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.296', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803224412 ms)
2025-02-17 03:02:35.940 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803233760 ms)
2025-02-17 03:02:35.941 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803233760 ms)
2025-02-17 03:02:35.942 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:02:25.400008+00:00'} (T+803233761 ms)
2025-02-17 03:03:05.959 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803263778 ms)
2025-02-17 03:03:05.959 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803263779 ms)
2025-02-17 03:03:05.960 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48546)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803263779 ms)
2025-02-17 03:03:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803284413 ms)
2025-02-17 03:03:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803284413 ms)
2025-02-17 03:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:03:25.400157+00:00', 'runtime': '803280.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803284414 ms)
2025-02-17 03:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:02:25.401134+00:00', 'runtime': '803220.002', 'adc_vb_f': '12.292', 'adc_va_f': '1.517', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803284415 ms)
2025-02-17 03:03:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803293445 ms)
2025-02-17 03:03:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803293445 ms)
2025-02-17 03:03:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:03:25.400373+00:00'} (T+803293446 ms)
2025-02-17 03:04:17.091 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803334910 ms)
2025-02-17 03:04:17.091 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803334910 ms)
2025-02-17 03:04:17.092 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48622)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803334911 ms)
2025-02-17 03:04:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803344416 ms)
2025-02-17 03:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803344416 ms)
2025-02-17 03:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:04:25.400163+00:00', 'runtime': '803340.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803344417 ms)
2025-02-17 03:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:03:25.400157+00:00', 'runtime': '803280.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803344418 ms)
2025-02-17 03:04:35.665 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803353485 ms)
2025-02-17 03:04:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803353485 ms)
2025-02-17 03:04:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:04:25.400166+00:00'} (T+803353486 ms)
2025-02-17 03:05:21.260 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803399079 ms)
2025-02-17 03:05:21.260 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803399079 ms)
2025-02-17 03:05:21.261 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48692)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803399080 ms)
2025-02-17 03:05:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803404405 ms)
2025-02-17 03:05:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803404405 ms)
2025-02-17 03:05:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:05:25.400168+00:00', 'runtime': '803400.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803404406 ms)
2025-02-17 03:05:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:04:25.400163+00:00', 'runtime': '803340.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.5', 'ahl_t': '1585.5', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803404407 ms)
2025-02-17 03:05:35.757 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803413577 ms)
2025-02-17 03:05:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803413577 ms)
2025-02-17 03:05:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:05:25.400157+00:00'} (T+803413578 ms)
2025-02-17 03:06:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803464408 ms)
2025-02-17 03:06:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803464408 ms)
2025-02-17 03:06:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:06:25.400193+00:00', 'runtime': '803460.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803464409 ms)
2025-02-17 03:06:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:05:25.400168+00:00', 'runtime': '803400.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803464410 ms)
2025-02-17 03:06:35.720 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803473540 ms)
2025-02-17 03:06:35.721 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803473540 ms)
2025-02-17 03:06:35.722 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:06:25.400064+00:00'} (T+803473542 ms)
2025-02-17 03:07:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803524411 ms)
2025-02-17 03:07:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803524412 ms)
2025-02-17 03:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:07:25.400161+00:00', 'runtime': '803520.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803524413 ms)
2025-02-17 03:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:06:25.400193+00:00', 'runtime': '803460.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803524413 ms)
2025-02-17 03:07:35.752 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803533571 ms)
2025-02-17 03:07:35.752 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803533572 ms)
2025-02-17 03:07:35.754 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:07:25.400072+00:00'} (T+803533573 ms)
2025-02-17 03:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803584414 ms)
2025-02-17 03:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803584415 ms)
2025-02-17 03:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:08:25.400160+00:00', 'runtime': '803580.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803584416 ms)
2025-02-17 03:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:07:25.400161+00:00', 'runtime': '803520.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803584416 ms)
2025-02-17 03:08:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803593438 ms)
2025-02-17 03:08:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803593439 ms)
2025-02-17 03:08:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:08:25.400031+00:00'} (T+803593440 ms)
2025-02-17 03:09:18.631 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803636450 ms)
2025-02-17 03:09:18.631 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803636451 ms)
2025-02-17 03:09:18.632 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48956)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803636451 ms)
2025-02-17 03:09:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803644417 ms)
2025-02-17 03:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803644418 ms)
2025-02-17 03:09:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:09:25.400252+00:00', 'runtime': '803640.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803644419 ms)
2025-02-17 03:09:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:08:25.400160+00:00', 'runtime': '803580.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803644419 ms)
2025-02-17 03:09:35.886 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803653705 ms)
2025-02-17 03:09:35.886 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803653705 ms)
2025-02-17 03:09:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:09:25.400254+00:00'} (T+803653706 ms)
2025-02-17 03:10:03.769 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803681588 ms)
2025-02-17 03:10:03.769 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803681588 ms)
2025-02-17 03:10:03.770 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49006)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803681589 ms)
2025-02-17 03:10:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803704405 ms)
2025-02-17 03:10:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803704405 ms)
2025-02-17 03:10:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:10:25.400263+00:00', 'runtime': '803700.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803704406 ms)
2025-02-17 03:10:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:09:25.400252+00:00', 'runtime': '803640.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803704407 ms)
2025-02-17 03:10:36.057 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803713876 ms)
2025-02-17 03:10:36.057 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803713877 ms)
2025-02-17 03:10:36.059 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:10:25.400438+00:00'} (T+803713878 ms)
2025-02-17 03:11:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803764408 ms)
2025-02-17 03:11:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803764409 ms)
2025-02-17 03:11:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:11:25.400193+00:00', 'runtime': '803760.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803764409 ms)
2025-02-17 03:11:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:10:25.400263+00:00', 'runtime': '803700.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803764410 ms)
2025-02-17 03:12:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803824415 ms)
2025-02-17 03:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803824415 ms)
2025-02-17 03:12:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:12:25.400244+00:00', 'runtime': '803820.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.002', 'adc_il_f': '0.476', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803824416 ms)
2025-02-17 03:12:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:11:25.400193+00:00', 'runtime': '803760.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.52', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803824417 ms)
2025-02-17 03:13:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803884417 ms)
2025-02-17 03:13:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803884417 ms)
2025-02-17 03:13:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:13:25.400161+00:00', 'runtime': '803880.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.526', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803884418 ms)
2025-02-17 03:13:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:12:25.400244+00:00', 'runtime': '803820.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.002', 'adc_il_f': '0.476', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803884418 ms)
2025-02-17 03:13:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803893430 ms)
2025-02-17 03:13:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803893430 ms)
2025-02-17 03:13:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:13:25.400333+00:00'} (T+803893431 ms)
2025-02-17 03:14:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+803944422 ms)
2025-02-17 03:14:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+803944423 ms)
2025-02-17 03:14:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:14:25.400169+00:00', 'runtime': '803940.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.529', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803944424 ms)
2025-02-17 03:14:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:13:25.400161+00:00', 'runtime': '803880.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.526', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+803944424 ms)
2025-02-17 03:14:35.647 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803953466 ms)
2025-02-17 03:14:35.648 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+803953467 ms)
2025-02-17 03:14:35.649 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:14:25.400189+00:00'} (T+803953468 ms)
2025-02-17 03:14:56.209 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+803974028 ms)
2025-02-17 03:14:56.209 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+803974028 ms)
2025-02-17 03:14:56.210 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49318)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+803974029 ms)
2025-02-17 03:15:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804004407 ms)
2025-02-17 03:15:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804004407 ms)
2025-02-17 03:15:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:15:25.400160+00:00', 'runtime': '804000.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.538', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804004408 ms)
2025-02-17 03:15:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:14:25.400169+00:00', 'runtime': '803940.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.529', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804004409 ms)
2025-02-17 03:15:36.207 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804014027 ms)
2025-02-17 03:15:36.208 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804014027 ms)
2025-02-17 03:15:36.209 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:15:25.400160+00:00'} (T+804014028 ms)
2025-02-17 03:16:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804064412 ms)
2025-02-17 03:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804064413 ms)
2025-02-17 03:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:16:25.400195+00:00', 'runtime': '804060.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.553', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804064414 ms)
2025-02-17 03:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:15:25.400160+00:00', 'runtime': '804000.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.538', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804064414 ms)
2025-02-17 03:16:36.542 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804074361 ms)
2025-02-17 03:16:36.542 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804074362 ms)
2025-02-17 03:16:36.543 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:16:25.400108+00:00'} (T+804074362 ms)
2025-02-17 03:17:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804124419 ms)
2025-02-17 03:17:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804124419 ms)
2025-02-17 03:17:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:17:25.400262+00:00', 'runtime': '804120.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804124420 ms)
2025-02-17 03:17:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:16:25.400195+00:00', 'runtime': '804060.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.553', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804124420 ms)
2025-02-17 03:18:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804184412 ms)
2025-02-17 03:18:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804184412 ms)
2025-02-17 03:18:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:18:25.400190+00:00', 'runtime': '804180.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804184413 ms)
2025-02-17 03:18:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:17:25.400262+00:00', 'runtime': '804120.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.6', 'ahl_t': '1585.6', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804184414 ms)
2025-02-17 03:19:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804244419 ms)
2025-02-17 03:19:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804244420 ms)
2025-02-17 03:19:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:19:25.400171+00:00', 'runtime': '804240.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804244421 ms)
2025-02-17 03:19:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:18:25.400190+00:00', 'runtime': '804180.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804244421 ms)
2025-02-17 03:19:29.625 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+804247444 ms)
2025-02-17 03:19:29.625 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+804247445 ms)
2025-02-17 03:19:29.626 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49616)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+804247445 ms)
2025-02-17 03:19:41.719 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+804259538 ms)
2025-02-17 03:19:41.719 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+804259538 ms)
2025-02-17 03:19:41.719 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49632)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+804259538 ms)
2025-02-17 03:20:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804304410 ms)
2025-02-17 03:20:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804304411 ms)
2025-02-17 03:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:20:25.400250+00:00', 'runtime': '804300.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.535', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804304412 ms)
2025-02-17 03:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:19:25.400171+00:00', 'runtime': '804240.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804304412 ms)
2025-02-17 03:21:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804364419 ms)
2025-02-17 03:21:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804364420 ms)
2025-02-17 03:21:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:21:25.400183+00:00', 'runtime': '804360.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.002', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804364421 ms)
2025-02-17 03:21:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:20:25.400250+00:00', 'runtime': '804300.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.535', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804364422 ms)
2025-02-17 03:22:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804424410 ms)
2025-02-17 03:22:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804424410 ms)
2025-02-17 03:22:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:22:25.400175+00:00', 'runtime': '804420.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804424411 ms)
2025-02-17 03:22:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:21:25.400183+00:00', 'runtime': '804360.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.002', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804424411 ms)
2025-02-17 03:22:35.631 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804433450 ms)
2025-02-17 03:22:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804433451 ms)
2025-02-17 03:22:35.632 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:22:25.400328+00:00'} (T+804433452 ms)
2025-02-17 03:23:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804484414 ms)
2025-02-17 03:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804484415 ms)
2025-02-17 03:23:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:23:25.400178+00:00', 'runtime': '804480.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.526', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804484415 ms)
2025-02-17 03:23:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:22:25.400175+00:00', 'runtime': '804420.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.523', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804484416 ms)
2025-02-17 03:23:41.123 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+804498943 ms)
2025-02-17 03:23:41.124 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+804498943 ms)
2025-02-17 03:23:41.124 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49884)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+804498943 ms)
2025-02-17 03:24:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804544404 ms)
2025-02-17 03:24:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804544404 ms)
2025-02-17 03:24:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:24:25.400171+00:00', 'runtime': '804540.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.55', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804544405 ms)
2025-02-17 03:24:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:23:25.400178+00:00', 'runtime': '804480.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.526', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804544406 ms)
2025-02-17 03:25:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804604411 ms)
2025-02-17 03:25:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804604411 ms)
2025-02-17 03:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:25:25.400182+00:00', 'runtime': '804600.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804604412 ms)
2025-02-17 03:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:24:25.400171+00:00', 'runtime': '804540.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.55', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804604412 ms)
2025-02-17 03:25:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804613426 ms)
2025-02-17 03:25:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804613427 ms)
2025-02-17 03:25:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:25:25.400342+00:00'} (T+804613428 ms)
2025-02-17 03:25:36.359 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+804614178 ms)
2025-02-17 03:25:36.359 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+804614178 ms)
2025-02-17 03:25:36.359 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50006)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+804614179 ms)
2025-02-17 03:26:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804664416 ms)
2025-02-17 03:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804664416 ms)
2025-02-17 03:26:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:26:25.402847+00:00', 'runtime': '804660.003', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804664417 ms)
2025-02-17 03:26:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:25:25.400182+00:00', 'runtime': '804600.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804664418 ms)
2025-02-17 03:26:35.741 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804673560 ms)
2025-02-17 03:26:35.741 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804673561 ms)
2025-02-17 03:26:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:26:25.400270+00:00'} (T+804673562 ms)
2025-02-17 03:26:52.454 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+804690273 ms)
2025-02-17 03:26:52.454 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+804690273 ms)
2025-02-17 03:26:52.454 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+804690273 ms)
2025-02-17 03:26:52.455 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+804690274 ms)
2025-02-17 03:26:52.455 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+804690274 ms)
2025-02-17 03:26:52.456 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+804690275 ms)
2025-02-17 03:27:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804724420 ms)
2025-02-17 03:27:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804724421 ms)
2025-02-17 03:27:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:27:25.402695+00:00', 'runtime': '804720.003', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804724422 ms)
2025-02-17 03:27:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:26:25.402847+00:00', 'runtime': '804660.003', 'adc_vb_f': '12.292', 'adc_va_f': '1.562', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804724422 ms)
2025-02-17 03:27:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804733428 ms)
2025-02-17 03:27:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+804733429 ms)
2025-02-17 03:27:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:27:25.400069+00:00'} (T+804733430 ms)
2025-02-17 03:27:46.597 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+804744416 ms)
2025-02-17 03:27:46.597 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+804744416 ms)
2025-02-17 03:27:46.597 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50156)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+804744417 ms)
2025-02-17 03:28:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804784409 ms)
2025-02-17 03:28:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804784409 ms)
2025-02-17 03:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:28:25.400249+00:00', 'runtime': '804780.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804784411 ms)
2025-02-17 03:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:27:25.402695+00:00', 'runtime': '804720.003', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804784412 ms)
2025-02-17 03:29:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804844415 ms)
2025-02-17 03:29:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804844415 ms)
2025-02-17 03:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:29:25.400163+00:00', 'runtime': '804840.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.572', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804844416 ms)
2025-02-17 03:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:28:25.400249+00:00', 'runtime': '804780.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.292', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804844417 ms)
2025-02-17 03:30:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804904405 ms)
2025-02-17 03:30:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804904405 ms)
2025-02-17 03:30:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:30:25.400592+00:00', 'runtime': '804900.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.575', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804904406 ms)
2025-02-17 03:30:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:29:25.400163+00:00', 'runtime': '804840.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.572', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804904407 ms)
2025-02-17 03:31:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+804964411 ms)
2025-02-17 03:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+804964412 ms)
2025-02-17 03:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:31:25.400194+00:00', 'runtime': '804960.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804964413 ms)
2025-02-17 03:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:30:25.400592+00:00', 'runtime': '804900.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.575', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.7', 'ahl_t': '1585.7', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+804964413 ms)
2025-02-17 03:32:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805024417 ms)
2025-02-17 03:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805024418 ms)
2025-02-17 03:32:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:32:25.400184+00:00', 'runtime': '805020.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805024419 ms)
2025-02-17 03:32:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:31:25.400194+00:00', 'runtime': '804960.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805024419 ms)
2025-02-17 03:33:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805084406 ms)
2025-02-17 03:33:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805084406 ms)
2025-02-17 03:33:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:33:25.400620+00:00', 'runtime': '805080.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805084407 ms)
2025-02-17 03:33:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:32:25.400184+00:00', 'runtime': '805020.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805084408 ms)
2025-02-17 03:33:49.093 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+805106912 ms)
2025-02-17 03:33:49.093 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+805106913 ms)
2025-02-17 03:33:49.094 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50538)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+805106913 ms)
2025-02-17 03:34:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805144409 ms)
2025-02-17 03:34:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805144410 ms)
2025-02-17 03:34:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:34:25.402744+00:00', 'runtime': '805140.003', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805144411 ms)
2025-02-17 03:34:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:33:25.400620+00:00', 'runtime': '805080.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805144412 ms)
2025-02-17 03:34:33.475 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+805151295 ms)
2025-02-17 03:35:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805204414 ms)
2025-02-17 03:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805204414 ms)
2025-02-17 03:35:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:35:25.401054+00:00', 'runtime': '805200.002', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.286', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805204416 ms)
2025-02-17 03:35:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:34:25.402744+00:00', 'runtime': '805140.003', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.289', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805204417 ms)
2025-02-17 03:35:53.311 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+805231130 ms)
2025-02-17 03:35:53.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+805231130 ms)
2025-02-17 03:35:53.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50678)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+805231131 ms)
2025-02-17 03:36:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805264417 ms)
2025-02-17 03:36:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805264418 ms)
2025-02-17 03:36:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:36:25.400612+00:00', 'runtime': '805260.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805264420 ms)
2025-02-17 03:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:35:25.401054+00:00', 'runtime': '805200.002', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.286', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805264420 ms)
2025-02-17 03:37:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805324421 ms)
2025-02-17 03:37:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805324421 ms)
2025-02-17 03:37:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:37:25.400248+00:00', 'runtime': '805320.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805324423 ms)
2025-02-17 03:37:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:36:25.400612+00:00', 'runtime': '805260.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805324424 ms)
2025-02-17 03:37:36.514 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+805334334 ms)
2025-02-17 03:37:36.515 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+805334334 ms)
2025-02-17 03:37:36.515 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50786)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+805334334 ms)
2025-02-17 03:38:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805384411 ms)
2025-02-17 03:38:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805384412 ms)
2025-02-17 03:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:38:25.400169+00:00', 'runtime': '805380.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805384414 ms)
2025-02-17 03:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:37:25.400248+00:00', 'runtime': '805320.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805384414 ms)
2025-02-17 03:39:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805444417 ms)
2025-02-17 03:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805444417 ms)
2025-02-17 03:39:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:39:25.400174+00:00', 'runtime': '805440.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805444419 ms)
2025-02-17 03:39:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:38:25.400169+00:00', 'runtime': '805380.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.566', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805444420 ms)
2025-02-17 03:40:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805504419 ms)
2025-02-17 03:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805504419 ms)
2025-02-17 03:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:40:25.402909+00:00', 'runtime': '805500.003', 'adc_vb_f': '12.289', 'adc_va_f': '1.578', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805504422 ms)
2025-02-17 03:40:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:39:25.400174+00:00', 'runtime': '805440.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.569', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805504422 ms)
2025-02-17 03:40:48.761 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+805526580 ms)
2025-02-17 03:40:48.761 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+805526580 ms)
2025-02-17 03:40:48.762 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+805526581 ms)
2025-02-17 03:41:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805564410 ms)
2025-02-17 03:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805564411 ms)
2025-02-17 03:41:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:41:25.400818+00:00', 'runtime': '805560.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.581', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805564413 ms)
2025-02-17 03:41:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:40:25.402909+00:00', 'runtime': '805500.003', 'adc_vb_f': '12.289', 'adc_va_f': '1.578', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805564413 ms)
2025-02-17 03:42:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805624416 ms)
2025-02-17 03:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805624417 ms)
2025-02-17 03:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:42:25.400191+00:00', 'runtime': '805620.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.59', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805624419 ms)
2025-02-17 03:42:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:41:25.400818+00:00', 'runtime': '805560.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.581', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805624419 ms)
2025-02-17 03:42:35.440 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+805633259 ms)
2025-02-17 03:42:35.440 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+805633260 ms)
2025-02-17 03:42:35.441 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:42:25.413052+00:00'} (T+805633261 ms)
2025-02-17 03:43:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805684423 ms)
2025-02-17 03:43:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805684423 ms)
2025-02-17 03:43:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:43:25.400170+00:00', 'runtime': '805680.001', 'adc_vb_f': '12.289', 'adc_va_f': '1.599', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805684425 ms)
2025-02-17 03:43:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:42:25.400191+00:00', 'runtime': '805620.001', 'adc_vb_f': '12.292', 'adc_va_f': '1.59', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805684426 ms)
2025-02-17 03:43:35.440 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+805693259 ms)
2025-02-17 03:43:35.440 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+805693260 ms)
2025-02-17 03:43:35.441 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:43:25.413125+00:00'} (T+805693261 ms)
2025-02-17 03:44:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805744411 ms)
2025-02-17 03:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805744411 ms)
2025-02-17 03:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:44:25.400241+00:00', 'runtime': '805740.001', 'adc_vb_f': '12.283', 'adc_va_f': '1.605', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.286', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805744415 ms)
2025-02-17 03:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:43:25.400170+00:00', 'runtime': '805680.001', 'adc_vb_f': '12.289', 'adc_va_f': '1.599', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.8', 'ahl_t': '1585.8', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805744416 ms)
2025-02-17 03:45:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805804415 ms)
2025-02-17 03:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805804415 ms)
2025-02-17 03:45:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:45:25.400897+00:00', 'runtime': '805800.001', 'adc_vb_f': '12.286', 'adc_va_f': '1.608', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.286', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805804420 ms)
2025-02-17 03:45:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:44:25.400241+00:00', 'runtime': '805740.001', 'adc_vb_f': '12.283', 'adc_va_f': '1.605', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.286', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805804420 ms)
2025-02-17 03:46:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805864402 ms)
2025-02-17 03:46:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805864403 ms)
2025-02-17 03:46:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:46:25.400594+00:00', 'runtime': '805860.001', 'adc_vb_f': '12.28', 'adc_va_f': '1.605', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805864406 ms)
2025-02-17 03:46:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:45:25.400897+00:00', 'runtime': '805800.001', 'adc_vb_f': '12.286', 'adc_va_f': '1.608', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.286', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.277', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805864407 ms)
2025-02-17 03:47:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805924408 ms)
2025-02-17 03:47:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805924408 ms)
2025-02-17 03:47:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:47:25.400211+00:00', 'runtime': '805920.001', 'adc_vb_f': '12.28', 'adc_va_f': '1.605', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805924412 ms)
2025-02-17 03:47:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:46:25.400594+00:00', 'runtime': '805860.001', 'adc_vb_f': '12.28', 'adc_va_f': '1.605', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805924413 ms)
2025-02-17 03:47:35.750 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+805933569 ms)
2025-02-17 03:47:35.751 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+805933570 ms)
2025-02-17 03:47:35.752 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:47:25.405066+00:00'} (T+805933571 ms)
2025-02-17 03:48:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+805984412 ms)
2025-02-17 03:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+805984412 ms)
2025-02-17 03:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:48:25.400048+00:00', 'runtime': '805980.001', 'adc_vb_f': '12.283', 'adc_va_f': '1.599', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805984417 ms)
2025-02-17 03:48:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:47:25.400211+00:00', 'runtime': '805920.001', 'adc_vb_f': '12.28', 'adc_va_f': '1.605', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+805984417 ms)
2025-02-17 03:48:52.439 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+806010258 ms)
2025-02-17 03:48:52.439 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+806010258 ms)
2025-02-17 03:48:52.440 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51508)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806010259 ms)
2025-02-17 03:49:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806044415 ms)
2025-02-17 03:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806044416 ms)
2025-02-17 03:49:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:49:25.400151+00:00', 'runtime': '806040.001', 'adc_vb_f': '12.289', 'adc_va_f': '1.599', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806044417 ms)
2025-02-17 03:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:48:25.400048+00:00', 'runtime': '805980.001', 'adc_vb_f': '12.283', 'adc_va_f': '1.599', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806044417 ms)
2025-02-17 03:50:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806104401 ms)
2025-02-17 03:50:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806104402 ms)
2025-02-17 03:50:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:50:25.400579+00:00', 'runtime': '806100.001', 'adc_vb_f': '12.286', 'adc_va_f': '1.599', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806104403 ms)
2025-02-17 03:50:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:49:25.400151+00:00', 'runtime': '806040.001', 'adc_vb_f': '12.289', 'adc_va_f': '1.599', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806104403 ms)
2025-02-17 03:51:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806164421 ms)
2025-02-17 03:51:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806164422 ms)
2025-02-17 03:51:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:51:25.402800+00:00', 'runtime': '806160.003', 'adc_vb_f': '12.28', 'adc_va_f': '1.602', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806164423 ms)
2025-02-17 03:51:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:50:25.400579+00:00', 'runtime': '806100.001', 'adc_vb_f': '12.286', 'adc_va_f': '1.599', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806164423 ms)
2025-02-17 03:51:35.918 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806173738 ms)
2025-02-17 03:51:35.919 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806173738 ms)
2025-02-17 03:51:35.920 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:51:25.400158+00:00'} (T+806173739 ms)
2025-02-17 03:52:05.758 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+806203577 ms)
2025-02-17 03:52:05.758 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+806203577 ms)
2025-02-17 03:52:05.758 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51710)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806203578 ms)
2025-02-17 03:52:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806224409 ms)
2025-02-17 03:52:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806224409 ms)
2025-02-17 03:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:52:25.400242+00:00', 'runtime': '806220.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.599', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806224411 ms)
2025-02-17 03:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:51:25.402800+00:00', 'runtime': '806160.003', 'adc_vb_f': '12.28', 'adc_va_f': '1.602', 'adc_vl_f': '12.28', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806224411 ms)
2025-02-17 03:52:37.868 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+806235688 ms)
2025-02-17 03:52:37.869 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+806235688 ms)
2025-02-17 03:52:37.869 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51746)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806235689 ms)
2025-02-17 03:53:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806284412 ms)
2025-02-17 03:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806284413 ms)
2025-02-17 03:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:53:25.400193+00:00', 'runtime': '806280.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.599', 'adc_vl_f': '12.274', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.271', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806284415 ms)
2025-02-17 03:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:52:25.400242+00:00', 'runtime': '806220.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.599', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.274', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806284415 ms)
2025-02-17 03:54:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806344416 ms)
2025-02-17 03:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806344416 ms)
2025-02-17 03:54:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:54:25.400188+00:00', 'runtime': '806340.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.593', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.271', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806344417 ms)
2025-02-17 03:54:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:53:25.400193+00:00', 'runtime': '806280.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.599', 'adc_vl_f': '12.274', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.28', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20110', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.271', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806344418 ms)
2025-02-17 03:54:43.111 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+806360930 ms)
2025-02-17 03:54:43.111 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+806360931 ms)
2025-02-17 03:54:43.112 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51876)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806360931 ms)
2025-02-17 03:55:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806404405 ms)
2025-02-17 03:55:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806404405 ms)
2025-02-17 03:55:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:55:25.400214+00:00', 'runtime': '806400.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.587', 'adc_vl_f': '12.265', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.271', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806404406 ms)
2025-02-17 03:55:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:54:25.400188+00:00', 'runtime': '806340.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.593', 'adc_vl_f': '12.277', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.271', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806404407 ms)
2025-02-17 03:55:35.238 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+806413058 ms)
2025-02-17 03:55:35.239 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+806413058 ms)
2025-02-17 03:55:35.239 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51934)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806413058 ms)
2025-02-17 03:55:35.792 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806413611 ms)
2025-02-17 03:55:35.793 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806413612 ms)
2025-02-17 03:55:35.794 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 03:55:25.400325+00:00'} (T+806413613 ms)
2025-02-17 03:56:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806464407 ms)
2025-02-17 03:56:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806464407 ms)
2025-02-17 03:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:56:25.400209+00:00', 'runtime': '806460.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.584', 'adc_vl_f': '12.256', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.265', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806464408 ms)
2025-02-17 03:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:55:25.400214+00:00', 'runtime': '806400.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.587', 'adc_vl_f': '12.265', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.271', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806464409 ms)
2025-02-17 03:57:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806524411 ms)
2025-02-17 03:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806524412 ms)
2025-02-17 03:57:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:57:25.400217+00:00', 'runtime': '806520.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.578', 'adc_vl_f': '12.265', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806524413 ms)
2025-02-17 03:57:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:56:25.400209+00:00', 'runtime': '806460.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.584', 'adc_vl_f': '12.256', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1585.9', 'ahl_t': '1585.9', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.265', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806524413 ms)
2025-02-17 03:58:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806584415 ms)
2025-02-17 03:58:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806584415 ms)
2025-02-17 03:58:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:58:25.400178+00:00', 'runtime': '806580.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.572', 'adc_vl_f': '12.271', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806584416 ms)
2025-02-17 03:58:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:57:25.400217+00:00', 'runtime': '806520.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.578', 'adc_vl_f': '12.265', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806584417 ms)
2025-02-17 03:59:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806644404 ms)
2025-02-17 03:59:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806644405 ms)
2025-02-17 03:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 03:59:25.400191+00:00', 'runtime': '806640.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.569', 'adc_vl_f': '12.271', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806644406 ms)
2025-02-17 03:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:58:25.400178+00:00', 'runtime': '806580.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.572', 'adc_vl_f': '12.271', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806644407 ms)
2025-02-17 03:59:58.659 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+806676478 ms)
2025-02-17 03:59:58.659 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+806676478 ms)
2025-02-17 03:59:58.659 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52222)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806676479 ms)
2025-02-17 04:00:20.685 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+806698505 ms)
2025-02-17 04:00:20.686 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+806698505 ms)
2025-02-17 04:00:20.686 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806698505 ms)
2025-02-17 04:00:20.687 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+806698506 ms)
2025-02-17 04:00:20.687 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+806698506 ms)
2025-02-17 04:00:20.687 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806698507 ms)
2025-02-17 04:00:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806704409 ms)
2025-02-17 04:00:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806704410 ms)
2025-02-17 04:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:00:25.400220+00:00', 'runtime': '806700.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.566', 'adc_vl_f': '12.25', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806704412 ms)
2025-02-17 04:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 03:59:25.400191+00:00', 'runtime': '806640.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.569', 'adc_vl_f': '12.271', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806704412 ms)
2025-02-17 04:01:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806764411 ms)
2025-02-17 04:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806764411 ms)
2025-02-17 04:01:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:01:25.400618+00:00', 'runtime': '806760.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.566', 'adc_vl_f': '12.271', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806764413 ms)
2025-02-17 04:01:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:00:25.400220+00:00', 'runtime': '806700.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.566', 'adc_vl_f': '12.25', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.259', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806764413 ms)
2025-02-17 04:02:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806824417 ms)
2025-02-17 04:02:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806824417 ms)
2025-02-17 04:02:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:02:25.400661+00:00', 'runtime': '806820.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.566', 'adc_vl_f': '12.259', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806824419 ms)
2025-02-17 04:02:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:01:25.400618+00:00', 'runtime': '806760.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.566', 'adc_vl_f': '12.271', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806824420 ms)
2025-02-17 04:03:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806884406 ms)
2025-02-17 04:03:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806884406 ms)
2025-02-17 04:03:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:03:25.400176+00:00', 'runtime': '806880.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.559', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806884408 ms)
2025-02-17 04:03:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:02:25.400661+00:00', 'runtime': '806820.001', 'adc_vb_f': '12.277', 'adc_va_f': '1.566', 'adc_vl_f': '12.259', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.277', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806884409 ms)
2025-02-17 04:03:26.985 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+806884805 ms)
2025-02-17 04:03:26.986 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+806884805 ms)
2025-02-17 04:03:26.986 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52440)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+806884805 ms)
2025-02-17 04:03:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806893434 ms)
2025-02-17 04:03:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806893434 ms)
2025-02-17 04:03:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:03:25.400330+00:00'} (T+806893435 ms)
2025-02-17 04:04:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+806944407 ms)
2025-02-17 04:04:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+806944407 ms)
2025-02-17 04:04:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:04:25.400194+00:00', 'runtime': '806940.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.556', 'adc_vl_f': '12.244', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806944409 ms)
2025-02-17 04:04:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:03:25.400176+00:00', 'runtime': '806880.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.559', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+806944410 ms)
2025-02-17 04:04:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806953429 ms)
2025-02-17 04:04:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+806953429 ms)
2025-02-17 04:04:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:04:25.400191+00:00'} (T+806953430 ms)
2025-02-17 04:05:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807004411 ms)
2025-02-17 04:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807004411 ms)
2025-02-17 04:05:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:05:25.400183+00:00', 'runtime': '807000.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.559', 'adc_vl_f': '12.244', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807004412 ms)
2025-02-17 04:05:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:04:25.400194+00:00', 'runtime': '806940.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.556', 'adc_vl_f': '12.244', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807004413 ms)
2025-02-17 04:05:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807013431 ms)
2025-02-17 04:05:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807013431 ms)
2025-02-17 04:05:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:05:25.400199+00:00'} (T+807013432 ms)
2025-02-17 04:06:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807064416 ms)
2025-02-17 04:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807064416 ms)
2025-02-17 04:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:06:25.400268+00:00', 'runtime': '807060.001', 'adc_vb_f': '12.268', 'adc_va_f': '1.559', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.271', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807064417 ms)
2025-02-17 04:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:05:25.400183+00:00', 'runtime': '807000.001', 'adc_vb_f': '12.274', 'adc_va_f': '1.559', 'adc_vl_f': '12.244', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807064418 ms)
2025-02-17 04:06:35.727 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807073546 ms)
2025-02-17 04:06:35.727 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807073547 ms)
2025-02-17 04:06:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:06:25.400270+00:00'} (T+807073548 ms)
2025-02-17 04:07:03.332 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+807101151 ms)
2025-02-17 04:07:03.332 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+807101151 ms)
2025-02-17 04:07:03.332 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52684)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+807101151 ms)
2025-02-17 04:07:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807124404 ms)
2025-02-17 04:07:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807124405 ms)
2025-02-17 04:07:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:07:25.400185+00:00', 'runtime': '807120.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.566', 'adc_vl_f': '12.253', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.271', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807124406 ms)
2025-02-17 04:07:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:06:25.400268+00:00', 'runtime': '807060.001', 'adc_vb_f': '12.268', 'adc_va_f': '1.559', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.271', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807124406 ms)
2025-02-17 04:07:35.873 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807133692 ms)
2025-02-17 04:07:35.874 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807133693 ms)
2025-02-17 04:07:35.875 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:07:25.400080+00:00'} (T+807133694 ms)
2025-02-17 04:08:03.483 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+807161303 ms)
2025-02-17 04:08:03.484 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+807161303 ms)
2025-02-17 04:08:03.484 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52750)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+807161303 ms)
2025-02-17 04:08:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807184407 ms)
2025-02-17 04:08:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807184408 ms)
2025-02-17 04:08:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:08:25.400193+00:00', 'runtime': '807180.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.566', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.268', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807184409 ms)
2025-02-17 04:08:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:07:25.400185+00:00', 'runtime': '807120.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.566', 'adc_vl_f': '12.253', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.271', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807184409 ms)
2025-02-17 04:08:27.597 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+807185417 ms)
2025-02-17 04:08:27.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+807185417 ms)
2025-02-17 04:08:27.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52776)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+807185417 ms)
2025-02-17 04:08:35.726 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807193545 ms)
2025-02-17 04:08:35.726 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807193546 ms)
2025-02-17 04:08:35.727 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:08:25.400456+00:00'} (T+807193547 ms)
2025-02-17 04:09:24.767 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+807242586 ms)
2025-02-17 04:09:24.767 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+807242587 ms)
2025-02-17 04:09:24.768 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52838)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+807242587 ms)
2025-02-17 04:09:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807244413 ms)
2025-02-17 04:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807244413 ms)
2025-02-17 04:09:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:09:25.400184+00:00', 'runtime': '807240.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.566', 'adc_vl_f': '12.244', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.268', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807244414 ms)
2025-02-17 04:09:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:08:25.400193+00:00', 'runtime': '807180.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.566', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.268', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807244415 ms)
2025-02-17 04:09:35.915 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807253734 ms)
2025-02-17 04:09:35.915 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807253735 ms)
2025-02-17 04:09:35.916 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:09:25.400060+00:00'} (T+807253736 ms)
2025-02-17 04:10:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807304415 ms)
2025-02-17 04:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807304415 ms)
2025-02-17 04:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:10:25.400535+00:00', 'runtime': '807300.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.569', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.262', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807304416 ms)
2025-02-17 04:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:09:25.400184+00:00', 'runtime': '807240.001', 'adc_vb_f': '12.271', 'adc_va_f': '1.566', 'adc_vl_f': '12.244', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.268', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.0', 'ahl_t': '1586.0', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807304417 ms)
2025-02-17 04:11:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807364403 ms)
2025-02-17 04:11:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807364403 ms)
2025-02-17 04:11:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:11:25.400154+00:00', 'runtime': '807360.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.575', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.265', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807364404 ms)
2025-02-17 04:11:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:10:25.400535+00:00', 'runtime': '807300.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.569', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.262', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807364405 ms)
2025-02-17 04:12:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807424409 ms)
2025-02-17 04:12:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807424409 ms)
2025-02-17 04:12:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:12:25.400167+00:00', 'runtime': '807420.001', 'adc_vb_f': '12.259', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.265', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807424410 ms)
2025-02-17 04:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:11:25.400154+00:00', 'runtime': '807360.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.575', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.265', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807424411 ms)
2025-02-17 04:13:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807484408 ms)
2025-02-17 04:13:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807484408 ms)
2025-02-17 04:13:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:13:25.400190+00:00', 'runtime': '807480.001', 'adc_vb_f': '12.259', 'adc_va_f': '1.59', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.265', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807484409 ms)
2025-02-17 04:13:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:12:25.400167+00:00', 'runtime': '807420.001', 'adc_vb_f': '12.259', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.265', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807484410 ms)
2025-02-17 04:13:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807493432 ms)
2025-02-17 04:13:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807493433 ms)
2025-02-17 04:13:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:13:25.400341+00:00'} (T+807493434 ms)
2025-02-17 04:14:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807544411 ms)
2025-02-17 04:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807544411 ms)
2025-02-17 04:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:14:25.400195+00:00', 'runtime': '807540.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.262', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807544412 ms)
2025-02-17 04:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:13:25.400190+00:00', 'runtime': '807480.001', 'adc_vb_f': '12.259', 'adc_va_f': '1.59', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.265', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807544413 ms)
2025-02-17 04:14:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807553437 ms)
2025-02-17 04:14:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807553437 ms)
2025-02-17 04:14:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:14:25.400207+00:00'} (T+807553438 ms)
2025-02-17 04:15:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807604415 ms)
2025-02-17 04:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807604416 ms)
2025-02-17 04:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:15:25.400196+00:00', 'runtime': '807600.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.259', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807604417 ms)
2025-02-17 04:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:14:25.400195+00:00', 'runtime': '807540.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.262', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807604417 ms)
2025-02-17 04:15:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807613421 ms)
2025-02-17 04:15:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807613422 ms)
2025-02-17 04:15:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:15:25.400207+00:00'} (T+807613423 ms)
2025-02-17 04:16:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807664421 ms)
2025-02-17 04:16:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807664421 ms)
2025-02-17 04:16:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:16:25.400184+00:00', 'runtime': '807660.001', 'adc_vb_f': '12.259', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.259', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807664422 ms)
2025-02-17 04:16:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:15:25.400196+00:00', 'runtime': '807600.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.259', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807664422 ms)
2025-02-17 04:16:35.621 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807673440 ms)
2025-02-17 04:16:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807673441 ms)
2025-02-17 04:16:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:16:25.400202+00:00'} (T+807673442 ms)
2025-02-17 04:17:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807724406 ms)
2025-02-17 04:17:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807724406 ms)
2025-02-17 04:17:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:17:25.400447+00:00', 'runtime': '807720.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807724407 ms)
2025-02-17 04:17:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:16:25.400184+00:00', 'runtime': '807660.001', 'adc_vb_f': '12.259', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.259', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807724408 ms)
2025-02-17 04:17:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807733431 ms)
2025-02-17 04:17:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807733432 ms)
2025-02-17 04:17:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:17:25.400196+00:00'} (T+807733433 ms)
2025-02-17 04:18:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807784409 ms)
2025-02-17 04:18:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807784409 ms)
2025-02-17 04:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:18:25.400214+00:00', 'runtime': '807780.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807784410 ms)
2025-02-17 04:18:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:17:25.400447+00:00', 'runtime': '807720.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807784411 ms)
2025-02-17 04:18:29.456 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+807787275 ms)
2025-02-17 04:18:29.456 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+807787276 ms)
2025-02-17 04:18:29.457 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+807787276 ms)
2025-02-17 04:18:35.761 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807793581 ms)
2025-02-17 04:18:35.762 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807793581 ms)
2025-02-17 04:18:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:18:25.400229+00:00'} (T+807793583 ms)
2025-02-17 04:19:25.568 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+807843387 ms)
2025-02-17 04:19:25.568 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+807843388 ms)
2025-02-17 04:19:25.569 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53486)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+807843388 ms)
2025-02-17 04:19:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807844413 ms)
2025-02-17 04:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807844414 ms)
2025-02-17 04:19:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:19:25.400198+00:00', 'runtime': '807840.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807844415 ms)
2025-02-17 04:19:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:18:25.400214+00:00', 'runtime': '807780.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807844415 ms)
2025-02-17 04:19:35.880 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807853699 ms)
2025-02-17 04:19:35.880 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807853700 ms)
2025-02-17 04:19:35.882 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:19:25.400044+00:00'} (T+807853701 ms)
2025-02-17 04:20:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807904415 ms)
2025-02-17 04:20:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807904415 ms)
2025-02-17 04:20:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:20:25.400509+00:00', 'runtime': '807900.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807904416 ms)
2025-02-17 04:20:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:19:25.400198+00:00', 'runtime': '807840.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807904417 ms)
2025-02-17 04:20:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807913438 ms)
2025-02-17 04:20:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+807913438 ms)
2025-02-17 04:20:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:20:25.400447+00:00'} (T+807913439 ms)
2025-02-17 04:21:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+807964418 ms)
2025-02-17 04:21:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+807964419 ms)
2025-02-17 04:21:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:21:25.400178+00:00', 'runtime': '807960.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807964420 ms)
2025-02-17 04:21:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:20:25.400509+00:00', 'runtime': '807900.001', 'adc_vb_f': '12.256', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.256', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+807964421 ms)
2025-02-17 04:22:23.830 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808021649 ms)
2025-02-17 04:22:23.830 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808021649 ms)
2025-02-17 04:22:23.831 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53678)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808021650 ms)
2025-02-17 04:22:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808024402 ms)
2025-02-17 04:22:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808024403 ms)
2025-02-17 04:22:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:22:25.400163+00:00', 'runtime': '808020.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808024404 ms)
2025-02-17 04:22:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:21:25.400178+00:00', 'runtime': '807960.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808024404 ms)
2025-02-17 04:22:26.910 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808024729 ms)
2025-02-17 04:22:26.910 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808024730 ms)
2025-02-17 04:22:26.911 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53682)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808024730 ms)
2025-02-17 04:23:25.634 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808083453 ms)
2025-02-17 04:23:25.635 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808083454 ms)
2025-02-17 04:23:25.639 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:23:25.400177+00:00', 'runtime': '808080.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808083458 ms)
2025-02-17 04:23:25.639 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:22:25.400163+00:00', 'runtime': '808020.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.1', 'ahl_t': '1586.1', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808083458 ms)
2025-02-17 04:24:10.110 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808127929 ms)
2025-02-17 04:24:10.110 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808127929 ms)
2025-02-17 04:24:10.110 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53792)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808127930 ms)
2025-02-17 04:24:17.208 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808135028 ms)
2025-02-17 04:24:17.209 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808135028 ms)
2025-02-17 04:24:17.209 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53800)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808135028 ms)
2025-02-17 04:24:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808144405 ms)
2025-02-17 04:24:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808144406 ms)
2025-02-17 04:24:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:24:25.400132+00:00', 'runtime': '808140.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808144408 ms)
2025-02-17 04:24:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:23:25.400177+00:00', 'runtime': '808080.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808144408 ms)
2025-02-17 04:24:46.338 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808164158 ms)
2025-02-17 04:24:46.339 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808164158 ms)
2025-02-17 04:24:46.339 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53830)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808164158 ms)
2025-02-17 04:25:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808204404 ms)
2025-02-17 04:25:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808204404 ms)
2025-02-17 04:25:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:25:25.400186+00:00', 'runtime': '808200.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808204406 ms)
2025-02-17 04:25:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:24:25.400132+00:00', 'runtime': '808140.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808204407 ms)
2025-02-17 04:26:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808264407 ms)
2025-02-17 04:26:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808264407 ms)
2025-02-17 04:26:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:26:25.400206+00:00', 'runtime': '808260.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808264409 ms)
2025-02-17 04:26:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:25:25.400186+00:00', 'runtime': '808200.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808264409 ms)
2025-02-17 04:26:35.745 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808273564 ms)
2025-02-17 04:26:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808273565 ms)
2025-02-17 04:26:35.747 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:26:25.400346+00:00'} (T+808273566 ms)
2025-02-17 04:27:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808324408 ms)
2025-02-17 04:27:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808324408 ms)
2025-02-17 04:27:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:27:25.400178+00:00', 'runtime': '808320.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808324409 ms)
2025-02-17 04:27:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:26:25.400206+00:00', 'runtime': '808260.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808324410 ms)
2025-02-17 04:27:35.810 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808333629 ms)
2025-02-17 04:27:35.810 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808333630 ms)
2025-02-17 04:27:35.811 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:27:25.400052+00:00'} (T+808333630 ms)
2025-02-17 04:28:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808384423 ms)
2025-02-17 04:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808384423 ms)
2025-02-17 04:28:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:28:25.400542+00:00', 'runtime': '808380.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808384425 ms)
2025-02-17 04:28:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:27:25.400178+00:00', 'runtime': '808320.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808384426 ms)
2025-02-17 04:28:31.717 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808389536 ms)
2025-02-17 04:28:31.717 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808389536 ms)
2025-02-17 04:28:31.718 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54078)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808389537 ms)
2025-02-17 04:28:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808393433 ms)
2025-02-17 04:28:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808393433 ms)
2025-02-17 04:28:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:28:25.399991+00:00'} (T+808393434 ms)
2025-02-17 04:29:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808444407 ms)
2025-02-17 04:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808444407 ms)
2025-02-17 04:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:29:25.400168+00:00', 'runtime': '808440.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808444408 ms)
2025-02-17 04:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:28:25.400542+00:00', 'runtime': '808380.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808444408 ms)
2025-02-17 04:29:29.876 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808447695 ms)
2025-02-17 04:29:29.876 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808447696 ms)
2025-02-17 04:29:29.877 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54142)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808447696 ms)
2025-02-17 04:29:34.949 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808452768 ms)
2025-02-17 04:29:34.949 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808452769 ms)
2025-02-17 04:29:34.950 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54150)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808452769 ms)
2025-02-17 04:29:35.672 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808453491 ms)
2025-02-17 04:29:35.673 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808453492 ms)
2025-02-17 04:29:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:29:25.400201+00:00'} (T+808453493 ms)
2025-02-17 04:30:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808504406 ms)
2025-02-17 04:30:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808504406 ms)
2025-02-17 04:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:30:25.400210+00:00', 'runtime': '808500.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808504410 ms)
2025-02-17 04:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:29:25.400168+00:00', 'runtime': '808440.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808504411 ms)
2025-02-17 04:30:35.734 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808513553 ms)
2025-02-17 04:30:35.735 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808513554 ms)
2025-02-17 04:30:35.736 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:30:25.400126+00:00'} (T+808513555 ms)
2025-02-17 04:31:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808564406 ms)
2025-02-17 04:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808564407 ms)
2025-02-17 04:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:31:25.400186+00:00', 'runtime': '808560.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808564408 ms)
2025-02-17 04:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:30:25.400210+00:00', 'runtime': '808500.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808564408 ms)
2025-02-17 04:31:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808573433 ms)
2025-02-17 04:31:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808573433 ms)
2025-02-17 04:31:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:31:25.400069+00:00'} (T+808573434 ms)
2025-02-17 04:32:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808624406 ms)
2025-02-17 04:32:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808624407 ms)
2025-02-17 04:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:32:25.400184+00:00', 'runtime': '808620.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808624408 ms)
2025-02-17 04:32:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:31:25.400186+00:00', 'runtime': '808560.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808624408 ms)
2025-02-17 04:32:35.675 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808633494 ms)
2025-02-17 04:32:35.676 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808633495 ms)
2025-02-17 04:32:35.677 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:32:25.400186+00:00'} (T+808633496 ms)
2025-02-17 04:33:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808684420 ms)
2025-02-17 04:33:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808684421 ms)
2025-02-17 04:33:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:33:25.402779+00:00', 'runtime': '808680.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808684425 ms)
2025-02-17 04:33:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:32:25.400184+00:00', 'runtime': '808620.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808684425 ms)
2025-02-17 04:33:35.847 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808693666 ms)
2025-02-17 04:33:35.847 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808693667 ms)
2025-02-17 04:33:35.848 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:33:25.400197+00:00'} (T+808693668 ms)
2025-02-17 04:34:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808744422 ms)
2025-02-17 04:34:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808744423 ms)
2025-02-17 04:34:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:34:25.402772+00:00', 'runtime': '808740.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808744424 ms)
2025-02-17 04:34:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:33:25.402779+00:00', 'runtime': '808680.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808744424 ms)
2025-02-17 04:34:35.731 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808753551 ms)
2025-02-17 04:34:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808753551 ms)
2025-02-17 04:34:35.733 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:34:25.400204+00:00'} (T+808753552 ms)
2025-02-17 04:35:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808804407 ms)
2025-02-17 04:35:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808804407 ms)
2025-02-17 04:35:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:35:25.400179+00:00', 'runtime': '808800.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808804408 ms)
2025-02-17 04:35:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:34:25.402772+00:00', 'runtime': '808740.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808804409 ms)
2025-02-17 04:35:35.672 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808813491 ms)
2025-02-17 04:35:35.673 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808813492 ms)
2025-02-17 04:35:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:35:25.400071+00:00'} (T+808813493 ms)
2025-02-17 04:36:08.547 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808846367 ms)
2025-02-17 04:36:08.548 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808846367 ms)
2025-02-17 04:36:08.548 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54580)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808846367 ms)
2025-02-17 04:36:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808864409 ms)
2025-02-17 04:36:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808864409 ms)
2025-02-17 04:36:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:36:25.400180+00:00', 'runtime': '808860.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808864410 ms)
2025-02-17 04:36:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:35:25.400179+00:00', 'runtime': '808800.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.2', 'ahl_t': '1586.2', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808864411 ms)
2025-02-17 04:36:35.729 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808873548 ms)
2025-02-17 04:36:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808873549 ms)
2025-02-17 04:36:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:36:25.400119+00:00'} (T+808873550 ms)
2025-02-17 04:37:01.698 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808899517 ms)
2025-02-17 04:37:01.698 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808899517 ms)
2025-02-17 04:37:01.698 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54638)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808899518 ms)
2025-02-17 04:37:11.785 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+808909604 ms)
2025-02-17 04:37:11.785 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+808909604 ms)
2025-02-17 04:37:11.785 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54650)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+808909605 ms)
2025-02-17 04:37:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808924410 ms)
2025-02-17 04:37:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808924411 ms)
2025-02-17 04:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:37:25.400156+00:00', 'runtime': '808920.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808924412 ms)
2025-02-17 04:37:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:36:25.400180+00:00', 'runtime': '808860.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808924412 ms)
2025-02-17 04:37:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808933430 ms)
2025-02-17 04:37:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808933431 ms)
2025-02-17 04:37:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:37:25.400059+00:00'} (T+808933432 ms)
2025-02-17 04:38:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+808984413 ms)
2025-02-17 04:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+808984413 ms)
2025-02-17 04:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:38:25.400164+00:00', 'runtime': '808980.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808984414 ms)
2025-02-17 04:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:37:25.400156+00:00', 'runtime': '808920.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+808984414 ms)
2025-02-17 04:38:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808993430 ms)
2025-02-17 04:38:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+808993430 ms)
2025-02-17 04:38:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:38:25.400172+00:00'} (T+808993431 ms)
2025-02-17 04:39:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809044415 ms)
2025-02-17 04:39:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809044415 ms)
2025-02-17 04:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:39:25.400186+00:00', 'runtime': '809040.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809044417 ms)
2025-02-17 04:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:38:25.400164+00:00', 'runtime': '808980.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809044417 ms)
2025-02-17 04:39:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809053424 ms)
2025-02-17 04:39:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809053425 ms)
2025-02-17 04:39:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:39:25.400199+00:00'} (T+809053426 ms)
2025-02-17 04:40:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809104418 ms)
2025-02-17 04:40:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809104418 ms)
2025-02-17 04:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:40:25.400211+00:00', 'runtime': '809100.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809104419 ms)
2025-02-17 04:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:39:25.400186+00:00', 'runtime': '809040.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809104420 ms)
2025-02-17 04:40:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809113437 ms)
2025-02-17 04:40:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809113438 ms)
2025-02-17 04:40:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:40:25.400239+00:00'} (T+809113439 ms)
2025-02-17 04:41:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809164405 ms)
2025-02-17 04:41:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809164405 ms)
2025-02-17 04:41:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:41:25.400198+00:00', 'runtime': '809160.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809164409 ms)
2025-02-17 04:41:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:40:25.400211+00:00', 'runtime': '809100.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809164410 ms)
2025-02-17 04:41:35.787 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809173606 ms)
2025-02-17 04:41:35.788 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809173607 ms)
2025-02-17 04:41:35.789 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:41:25.400207+00:00'} (T+809173608 ms)
2025-02-17 04:41:42.201 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809180020 ms)
2025-02-17 04:41:42.201 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809180020 ms)
2025-02-17 04:41:42.202 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54938)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809180021 ms)
2025-02-17 04:42:17.354 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809215173 ms)
2025-02-17 04:42:17.354 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809215173 ms)
2025-02-17 04:42:17.354 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54976)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809215174 ms)
2025-02-17 04:42:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809224407 ms)
2025-02-17 04:42:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809224408 ms)
2025-02-17 04:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:42:25.400177+00:00', 'runtime': '809220.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809224409 ms)
2025-02-17 04:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:41:25.400198+00:00', 'runtime': '809160.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809224411 ms)
2025-02-17 04:42:35.644 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809233463 ms)
2025-02-17 04:42:35.645 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809233464 ms)
2025-02-17 04:42:35.647 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:42:25.400027+00:00'} (T+809233466 ms)
2025-02-17 04:42:53.483 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809251303 ms)
2025-02-17 04:42:53.484 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809251303 ms)
2025-02-17 04:42:53.484 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55018)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809251303 ms)
2025-02-17 04:43:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809284411 ms)
2025-02-17 04:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809284411 ms)
2025-02-17 04:43:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:43:25.400163+00:00', 'runtime': '809280.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809284412 ms)
2025-02-17 04:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:42:25.400177+00:00', 'runtime': '809220.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809284413 ms)
2025-02-17 04:43:35.880 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809293699 ms)
2025-02-17 04:43:35.881 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809293700 ms)
2025-02-17 04:43:35.885 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:43:25.400145+00:00'} (T+809293704 ms)
2025-02-17 04:44:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809344411 ms)
2025-02-17 04:44:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809344412 ms)
2025-02-17 04:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:44:25.400160+00:00', 'runtime': '809340.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809344413 ms)
2025-02-17 04:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:43:25.400163+00:00', 'runtime': '809280.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809344413 ms)
2025-02-17 04:44:36.060 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809353879 ms)
2025-02-17 04:44:36.060 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809353880 ms)
2025-02-17 04:44:36.062 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:44:25.400788+00:00'} (T+809353881 ms)
2025-02-17 04:45:13.653 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+809391473 ms)
2025-02-17 04:45:13.654 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+809391473 ms)
2025-02-17 04:45:13.654 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809391473 ms)
2025-02-17 04:45:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809404414 ms)
2025-02-17 04:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809404415 ms)
2025-02-17 04:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:45:25.400272+00:00', 'runtime': '809400.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809404416 ms)
2025-02-17 04:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:44:25.400160+00:00', 'runtime': '809340.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809404416 ms)
2025-02-17 04:45:35.664 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809413484 ms)
2025-02-17 04:45:35.665 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809413484 ms)
2025-02-17 04:45:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:45:25.400281+00:00'} (T+809413486 ms)
2025-02-17 04:46:09.796 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809447615 ms)
2025-02-17 04:46:09.796 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809447616 ms)
2025-02-17 04:46:09.797 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55238)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809447616 ms)
2025-02-17 04:46:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809464415 ms)
2025-02-17 04:46:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809464416 ms)
2025-02-17 04:46:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:46:25.400168+00:00', 'runtime': '809460.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809464417 ms)
2025-02-17 04:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:45:25.400272+00:00', 'runtime': '809400.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809464417 ms)
2025-02-17 04:46:35.424 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809473243 ms)
2025-02-17 04:46:35.424 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809473244 ms)
2025-02-17 04:46:35.426 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:46:25.400147+00:00'} (T+809473245 ms)
2025-02-17 04:47:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809524416 ms)
2025-02-17 04:47:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809524417 ms)
2025-02-17 04:47:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:47:25.400204+00:00', 'runtime': '809520.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809524418 ms)
2025-02-17 04:47:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:46:25.400168+00:00', 'runtime': '809460.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809524418 ms)
2025-02-17 04:47:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809533433 ms)
2025-02-17 04:47:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809533433 ms)
2025-02-17 04:47:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:47:25.400398+00:00'} (T+809533434 ms)
2025-02-17 04:48:13.021 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809570840 ms)
2025-02-17 04:48:13.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809570840 ms)
2025-02-17 04:48:13.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55370)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809570841 ms)
2025-02-17 04:48:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809584419 ms)
2025-02-17 04:48:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809584419 ms)
2025-02-17 04:48:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:48:25.400172+00:00', 'runtime': '809580.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809584420 ms)
2025-02-17 04:48:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:47:25.400204+00:00', 'runtime': '809520.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809584421 ms)
2025-02-17 04:48:35.090 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809592909 ms)
2025-02-17 04:48:35.090 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809592909 ms)
2025-02-17 04:48:35.090 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55396)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809592909 ms)
2025-02-17 04:48:35.640 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809593459 ms)
2025-02-17 04:48:35.641 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809593460 ms)
2025-02-17 04:48:35.644 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:48:25.400186+00:00'} (T+809593463 ms)
2025-02-17 04:49:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809644418 ms)
2025-02-17 04:49:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809644419 ms)
2025-02-17 04:49:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:49:25.400172+00:00', 'runtime': '809640.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.59', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809644420 ms)
2025-02-17 04:49:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:48:25.400172+00:00', 'runtime': '809580.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.3', 'ahl_t': '1586.3', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809644420 ms)
2025-02-17 04:49:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809653438 ms)
2025-02-17 04:49:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809653438 ms)
2025-02-17 04:49:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:49:25.400165+00:00'} (T+809653439 ms)
2025-02-17 04:50:04.291 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809682110 ms)
2025-02-17 04:50:04.291 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809682110 ms)
2025-02-17 04:50:04.291 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55492)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809682110 ms)
2025-02-17 04:50:14.370 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+809692189 ms)
2025-02-17 04:50:14.370 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+809692189 ms)
2025-02-17 04:50:14.370 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55504)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+809692190 ms)
2025-02-17 04:50:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809704406 ms)
2025-02-17 04:50:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809704406 ms)
2025-02-17 04:50:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:50:25.400175+00:00', 'runtime': '809700.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809704407 ms)
2025-02-17 04:50:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:49:25.400172+00:00', 'runtime': '809640.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.59', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.253', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809704408 ms)
2025-02-17 04:50:35.909 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809713728 ms)
2025-02-17 04:50:35.910 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809713729 ms)
2025-02-17 04:50:35.911 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:50:25.400195+00:00'} (T+809713730 ms)
2025-02-17 04:51:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809764407 ms)
2025-02-17 04:51:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809764407 ms)
2025-02-17 04:51:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:51:25.400210+00:00', 'runtime': '809760.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809764408 ms)
2025-02-17 04:51:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:50:25.400175+00:00', 'runtime': '809700.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809764409 ms)
2025-02-17 04:52:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809824410 ms)
2025-02-17 04:52:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809824410 ms)
2025-02-17 04:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:52:25.400207+00:00', 'runtime': '809820.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809824411 ms)
2025-02-17 04:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:51:25.400210+00:00', 'runtime': '809760.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809824412 ms)
2025-02-17 04:53:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809884411 ms)
2025-02-17 04:53:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809884411 ms)
2025-02-17 04:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:53:25.400170+00:00', 'runtime': '809880.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809884412 ms)
2025-02-17 04:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:52:25.400207+00:00', 'runtime': '809820.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809884413 ms)
2025-02-17 04:54:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+809944414 ms)
2025-02-17 04:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+809944415 ms)
2025-02-17 04:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:54:25.400167+00:00', 'runtime': '809940.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809944416 ms)
2025-02-17 04:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:53:25.400170+00:00', 'runtime': '809880.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.593', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20111', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+809944416 ms)
2025-02-17 04:54:35.646 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809953465 ms)
2025-02-17 04:54:35.647 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+809953466 ms)
2025-02-17 04:54:35.648 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:54:25.400339+00:00'} (T+809953467 ms)
2025-02-17 04:55:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810004418 ms)
2025-02-17 04:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810004418 ms)
2025-02-17 04:55:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:55:25.400220+00:00', 'runtime': '810000.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810004419 ms)
2025-02-17 04:55:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:54:25.400167+00:00', 'runtime': '809940.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810004419 ms)
2025-02-17 04:55:36.157 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810013976 ms)
2025-02-17 04:55:36.157 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810013976 ms)
2025-02-17 04:55:36.158 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:55:25.400223+00:00'} (T+810013977 ms)
2025-02-17 04:56:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810064420 ms)
2025-02-17 04:56:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810064421 ms)
2025-02-17 04:56:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:56:25.400167+00:00', 'runtime': '810060.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810064422 ms)
2025-02-17 04:56:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:55:25.400220+00:00', 'runtime': '810000.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.25', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810064422 ms)
2025-02-17 04:56:30.881 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810068701 ms)
2025-02-17 04:56:30.882 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810068701 ms)
2025-02-17 04:56:30.882 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55912)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810068701 ms)
2025-02-17 04:57:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810124410 ms)
2025-02-17 04:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810124410 ms)
2025-02-17 04:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:57:25.400201+00:00', 'runtime': '810120.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810124411 ms)
2025-02-17 04:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:56:25.400167+00:00', 'runtime': '810060.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810124412 ms)
2025-02-17 04:58:19.096 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810176915 ms)
2025-02-17 04:58:19.096 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810176915 ms)
2025-02-17 04:58:19.096 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56028)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810176916 ms)
2025-02-17 04:58:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810184412 ms)
2025-02-17 04:58:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810184413 ms)
2025-02-17 04:58:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:58:25.400165+00:00', 'runtime': '810180.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810184414 ms)
2025-02-17 04:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:57:25.400201+00:00', 'runtime': '810120.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.596', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810184414 ms)
2025-02-17 04:59:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810244418 ms)
2025-02-17 04:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810244418 ms)
2025-02-17 04:59:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 04:59:25.400256+00:00', 'runtime': '810240.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810244422 ms)
2025-02-17 04:59:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:58:25.400165+00:00', 'runtime': '810180.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810244423 ms)
2025-02-17 04:59:35.630 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810253450 ms)
2025-02-17 04:59:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810253450 ms)
2025-02-17 04:59:35.632 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 04:59:25.413050+00:00'} (T+810253451 ms)
2025-02-17 05:00:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810304423 ms)
2025-02-17 05:00:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810304423 ms)
2025-02-17 05:00:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:00:25.402892+00:00', 'runtime': '810300.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810304425 ms)
2025-02-17 05:00:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 04:59:25.400256+00:00', 'runtime': '810240.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810304425 ms)
2025-02-17 05:00:35.878 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810313697 ms)
2025-02-17 05:00:35.878 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810313698 ms)
2025-02-17 05:00:35.879 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:00:25.400252+00:00'} (T+810313699 ms)
2025-02-17 05:01:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810364410 ms)
2025-02-17 05:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810364410 ms)
2025-02-17 05:01:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:01:25.402784+00:00', 'runtime': '810360.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.59', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810364414 ms)
2025-02-17 05:01:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:00:25.402892+00:00', 'runtime': '810300.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.241', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810364415 ms)
2025-02-17 05:01:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810373434 ms)
2025-02-17 05:01:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810373435 ms)
2025-02-17 05:01:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:01:25.400219+00:00'} (T+810373436 ms)
2025-02-17 05:02:17.467 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810415286 ms)
2025-02-17 05:02:17.467 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810415286 ms)
2025-02-17 05:02:17.467 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56282)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810415287 ms)
2025-02-17 05:02:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810424413 ms)
2025-02-17 05:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810424413 ms)
2025-02-17 05:02:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:02:25.400214+00:00', 'runtime': '810420.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.584', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810424415 ms)
2025-02-17 05:02:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:01:25.402784+00:00', 'runtime': '810360.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.59', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.4', 'ahl_t': '1586.4', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810424415 ms)
2025-02-17 05:02:35.634 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810433453 ms)
2025-02-17 05:02:35.634 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810433453 ms)
2025-02-17 05:02:35.635 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:02:25.400224+00:00'} (T+810433454 ms)
2025-02-17 05:03:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810484416 ms)
2025-02-17 05:03:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810484417 ms)
2025-02-17 05:03:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:03:25.400183+00:00', 'runtime': '810480.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.578', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810484418 ms)
2025-02-17 05:03:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:02:25.400214+00:00', 'runtime': '810420.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.584', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810484418 ms)
2025-02-17 05:03:46.646 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810504465 ms)
2025-02-17 05:03:46.646 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810504466 ms)
2025-02-17 05:03:46.647 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56378)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810504466 ms)
2025-02-17 05:04:08.762 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810526581 ms)
2025-02-17 05:04:08.762 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810526581 ms)
2025-02-17 05:04:08.762 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56404)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810526582 ms)
2025-02-17 05:04:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810544405 ms)
2025-02-17 05:04:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810544405 ms)
2025-02-17 05:04:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:04:25.400171+00:00', 'runtime': '810540.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.575', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810544406 ms)
2025-02-17 05:04:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:03:25.400183+00:00', 'runtime': '810480.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.578', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810544406 ms)
2025-02-17 05:05:12.897 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810590716 ms)
2025-02-17 05:05:12.897 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810590716 ms)
2025-02-17 05:05:12.897 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56472)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810590717 ms)
2025-02-17 05:05:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810604411 ms)
2025-02-17 05:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810604411 ms)
2025-02-17 05:05:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:05:25.400170+00:00', 'runtime': '810600.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.575', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810604412 ms)
2025-02-17 05:05:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:04:25.400171+00:00', 'runtime': '810540.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.575', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810604413 ms)
2025-02-17 05:06:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810664403 ms)
2025-02-17 05:06:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810664403 ms)
2025-02-17 05:06:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:06:25.400264+00:00', 'runtime': '810660.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.569', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810664404 ms)
2025-02-17 05:06:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:05:25.400170+00:00', 'runtime': '810600.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.575', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810664405 ms)
2025-02-17 05:07:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810724407 ms)
2025-02-17 05:07:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810724408 ms)
2025-02-17 05:07:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:07:25.400177+00:00', 'runtime': '810720.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.569', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810724409 ms)
2025-02-17 05:07:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:06:25.400264+00:00', 'runtime': '810660.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.569', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810724409 ms)
2025-02-17 05:08:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810784410 ms)
2025-02-17 05:08:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810784410 ms)
2025-02-17 05:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:08:25.402749+00:00', 'runtime': '810780.003', 'adc_vb_f': '12.25', 'adc_va_f': '1.578', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810784414 ms)
2025-02-17 05:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:07:25.400177+00:00', 'runtime': '810720.001', 'adc_vb_f': '12.253', 'adc_va_f': '1.569', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810784414 ms)
2025-02-17 05:08:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810793430 ms)
2025-02-17 05:08:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810793431 ms)
2025-02-17 05:08:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:08:25.400138+00:00'} (T+810793432 ms)
2025-02-17 05:09:10.271 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810828091 ms)
2025-02-17 05:09:10.272 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810828091 ms)
2025-02-17 05:09:10.272 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56732)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810828091 ms)
2025-02-17 05:09:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810844415 ms)
2025-02-17 05:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810844415 ms)
2025-02-17 05:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:09:25.402795+00:00', 'runtime': '810840.003', 'adc_vb_f': '12.25', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810844418 ms)
2025-02-17 05:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:08:25.402749+00:00', 'runtime': '810780.003', 'adc_vb_f': '12.25', 'adc_va_f': '1.578', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810844419 ms)
2025-02-17 05:09:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810853431 ms)
2025-02-17 05:09:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810853432 ms)
2025-02-17 05:09:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:09:25.400194+00:00'} (T+810853433 ms)
2025-02-17 05:10:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810904416 ms)
2025-02-17 05:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810904416 ms)
2025-02-17 05:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:10:25.402834+00:00', 'runtime': '810900.003', 'adc_vb_f': '12.247', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810904418 ms)
2025-02-17 05:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:09:25.402795+00:00', 'runtime': '810840.003', 'adc_vb_f': '12.25', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810904419 ms)
2025-02-17 05:10:28.414 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810906233 ms)
2025-02-17 05:10:28.414 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810906233 ms)
2025-02-17 05:10:28.414 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56816)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810906234 ms)
2025-02-17 05:10:35.744 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810913564 ms)
2025-02-17 05:10:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810913564 ms)
2025-02-17 05:10:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:10:25.400235+00:00'} (T+810913565 ms)
2025-02-17 05:11:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+810964418 ms)
2025-02-17 05:11:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+810964419 ms)
2025-02-17 05:11:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:11:25.402683+00:00', 'runtime': '810960.003', 'adc_vb_f': '12.247', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810964421 ms)
2025-02-17 05:11:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:10:25.402834+00:00', 'runtime': '810900.003', 'adc_vb_f': '12.247', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+810964421 ms)
2025-02-17 05:11:35.624 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810973443 ms)
2025-02-17 05:11:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+810973444 ms)
2025-02-17 05:11:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:11:25.400049+00:00'} (T+810973445 ms)
2025-02-17 05:11:42.583 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+810980402 ms)
2025-02-17 05:11:42.583 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+810980402 ms)
2025-02-17 05:11:42.583 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56898)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+810980402 ms)
2025-02-17 05:12:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811024406 ms)
2025-02-17 05:12:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811024406 ms)
2025-02-17 05:12:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:12:25.402800+00:00', 'runtime': '811020.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811024408 ms)
2025-02-17 05:12:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:11:25.402683+00:00', 'runtime': '810960.003', 'adc_vb_f': '12.247', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811024408 ms)
2025-02-17 05:12:35.629 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811033449 ms)
2025-02-17 05:12:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811033449 ms)
2025-02-17 05:12:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:12:25.400180+00:00'} (T+811033450 ms)
2025-02-17 05:13:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811084408 ms)
2025-02-17 05:13:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811084408 ms)
2025-02-17 05:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:13:25.401179+00:00', 'runtime': '811080.002', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811084410 ms)
2025-02-17 05:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:12:25.402800+00:00', 'runtime': '811020.003', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811084410 ms)
2025-02-17 05:13:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811093444 ms)
2025-02-17 05:13:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811093445 ms)
2025-02-17 05:13:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:13:25.400178+00:00'} (T+811093446 ms)
2025-02-17 05:14:20.863 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811138683 ms)
2025-02-17 05:14:20.864 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811138683 ms)
2025-02-17 05:14:20.864 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57068)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811138683 ms)
2025-02-17 05:14:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811144411 ms)
2025-02-17 05:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811144412 ms)
2025-02-17 05:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:14:25.402803+00:00', 'runtime': '811140.003', 'adc_vb_f': '12.25', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811144413 ms)
2025-02-17 05:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:13:25.401179+00:00', 'runtime': '811080.002', 'adc_vb_f': '12.253', 'adc_va_f': '1.587', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811144414 ms)
2025-02-17 05:14:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811153434 ms)
2025-02-17 05:14:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811153434 ms)
2025-02-17 05:14:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:14:25.400175+00:00'} (T+811153436 ms)
2025-02-17 05:15:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811204417 ms)
2025-02-17 05:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811204417 ms)
2025-02-17 05:15:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:15:25.402794+00:00', 'runtime': '811200.003', 'adc_vb_f': '12.247', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811204422 ms)
2025-02-17 05:15:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:14:25.402803+00:00', 'runtime': '811140.003', 'adc_vb_f': '12.25', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.5', 'ahl_t': '1586.5', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811204422 ms)
2025-02-17 05:16:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811264406 ms)
2025-02-17 05:16:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811264407 ms)
2025-02-17 05:16:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:16:25.400600+00:00', 'runtime': '811260.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811264409 ms)
2025-02-17 05:16:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:15:25.402794+00:00', 'runtime': '811200.003', 'adc_vb_f': '12.247', 'adc_va_f': '1.602', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811264409 ms)
2025-02-17 05:16:52.124 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811289943 ms)
2025-02-17 05:16:52.124 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811289943 ms)
2025-02-17 05:16:52.124 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57240)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811289944 ms)
2025-02-17 05:17:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811324412 ms)
2025-02-17 05:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811324413 ms)
2025-02-17 05:17:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:17:25.400193+00:00', 'runtime': '811320.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.596', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811324415 ms)
2025-02-17 05:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:16:25.400600+00:00', 'runtime': '811260.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811324415 ms)
2025-02-17 05:18:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811384401 ms)
2025-02-17 05:18:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811384401 ms)
2025-02-17 05:18:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:18:25.400201+00:00', 'runtime': '811380.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811384403 ms)
2025-02-17 05:18:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:17:25.400193+00:00', 'runtime': '811320.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.596', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811384404 ms)
2025-02-17 05:19:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811444406 ms)
2025-02-17 05:19:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811444406 ms)
2025-02-17 05:19:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:19:25.400198+00:00', 'runtime': '811440.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.605', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811444408 ms)
2025-02-17 05:19:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:18:25.400201+00:00', 'runtime': '811380.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.599', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811444408 ms)
2025-02-17 05:20:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811504409 ms)
2025-02-17 05:20:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811504410 ms)
2025-02-17 05:20:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:20:25.400173+00:00', 'runtime': '811500.001', 'adc_vb_f': '12.247', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811504412 ms)
2025-02-17 05:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:19:25.400198+00:00', 'runtime': '811440.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.605', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811504412 ms)
2025-02-17 05:21:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811564411 ms)
2025-02-17 05:21:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811564411 ms)
2025-02-17 05:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:21:25.400174+00:00', 'runtime': '811560.001', 'adc_vb_f': '12.247', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811564413 ms)
2025-02-17 05:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:20:25.400173+00:00', 'runtime': '811500.001', 'adc_vb_f': '12.247', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811564414 ms)
2025-02-17 05:21:43.587 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811581406 ms)
2025-02-17 05:21:43.587 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811581406 ms)
2025-02-17 05:21:43.587 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57544)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811581407 ms)
2025-02-17 05:22:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811624413 ms)
2025-02-17 05:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811624413 ms)
2025-02-17 05:22:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:22:25.400164+00:00', 'runtime': '811620.001', 'adc_vb_f': '12.25', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811624414 ms)
2025-02-17 05:22:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:21:25.400174+00:00', 'runtime': '811560.001', 'adc_vb_f': '12.247', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.002', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811624414 ms)
2025-02-17 05:22:29.727 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811627547 ms)
2025-02-17 05:22:29.728 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811627547 ms)
2025-02-17 05:22:29.728 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57594)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811627547 ms)
2025-02-17 05:23:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811684416 ms)
2025-02-17 05:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811684417 ms)
2025-02-17 05:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:23:25.400165+00:00', 'runtime': '811680.001', 'adc_vb_f': '12.25', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.244', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811684418 ms)
2025-02-17 05:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:22:25.400164+00:00', 'runtime': '811620.001', 'adc_vb_f': '12.25', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.238', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811684418 ms)
2025-02-17 05:24:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811744404 ms)
2025-02-17 05:24:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811744404 ms)
2025-02-17 05:24:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:24:25.400168+00:00', 'runtime': '811740.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.244', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811744405 ms)
2025-02-17 05:24:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:23:25.400165+00:00', 'runtime': '811680.001', 'adc_vb_f': '12.25', 'adc_va_f': '1.605', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.244', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811744406 ms)
2025-02-17 05:24:26.982 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811744802 ms)
2025-02-17 05:24:26.982 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811744802 ms)
2025-02-17 05:24:26.983 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57718)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811744802 ms)
2025-02-17 05:24:43.083 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811760903 ms)
2025-02-17 05:24:43.084 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811760903 ms)
2025-02-17 05:24:43.084 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57736)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811760903 ms)
2025-02-17 05:25:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811804405 ms)
2025-02-17 05:25:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811804406 ms)
2025-02-17 05:25:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:25:25.400173+00:00', 'runtime': '811800.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.244', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811804407 ms)
2025-02-17 05:25:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:24:25.400168+00:00', 'runtime': '811740.001', 'adc_vb_f': '12.244', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.244', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811804407 ms)
2025-02-17 05:26:25.286 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811863106 ms)
2025-02-17 05:26:25.287 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811863106 ms)
2025-02-17 05:26:25.287 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57856)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811863106 ms)
2025-02-17 05:26:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811864407 ms)
2025-02-17 05:26:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811864407 ms)
2025-02-17 05:26:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:26:25.400184+00:00', 'runtime': '811860.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.608', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811864408 ms)
2025-02-17 05:26:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:25:25.400173+00:00', 'runtime': '811800.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.244', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811864409 ms)
2025-02-17 05:27:10.432 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811908251 ms)
2025-02-17 05:27:10.432 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811908252 ms)
2025-02-17 05:27:10.433 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57904)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811908252 ms)
2025-02-17 05:27:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811924410 ms)
2025-02-17 05:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811924411 ms)
2025-02-17 05:27:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:27:25.400182+00:00', 'runtime': '811920.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811924412 ms)
2025-02-17 05:27:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:26:25.400184+00:00', 'runtime': '811860.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.608', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811924412 ms)
2025-02-17 05:27:31.559 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811929379 ms)
2025-02-17 05:27:31.560 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811929379 ms)
2025-02-17 05:27:31.560 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57926)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811929379 ms)
2025-02-17 05:27:33.562 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+811931381 ms)
2025-02-17 05:27:33.562 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+811931381 ms)
2025-02-17 05:27:33.563 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811931382 ms)
2025-02-17 05:27:33.564 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+811931383 ms)
2025-02-17 05:27:33.564 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+811931383 ms)
2025-02-17 05:27:33.564 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811931384 ms)
2025-02-17 05:27:33.565 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+811931385 ms)
2025-02-17 05:27:33.566 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+811931385 ms)
2025-02-17 05:27:33.566 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811931386 ms)
2025-02-17 05:27:33.567 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+811931387 ms)
2025-02-17 05:27:33.568 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+811931387 ms)
2025-02-17 05:27:33.568 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811931388 ms)
2025-02-17 05:27:33.572 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+811931391 ms)
2025-02-17 05:27:33.574 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+811931393 ms)
2025-02-17 05:27:33.577 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811931396 ms)
2025-02-17 05:27:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811933425 ms)
2025-02-17 05:27:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811933426 ms)
2025-02-17 05:27:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:27:25.400342+00:00'} (T+811933427 ms)
2025-02-17 05:28:05.703 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+811963522 ms)
2025-02-17 05:28:05.703 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+811963522 ms)
2025-02-17 05:28:05.704 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57966)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+811963523 ms)
2025-02-17 05:28:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+811984414 ms)
2025-02-17 05:28:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+811984415 ms)
2025-02-17 05:28:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:28:25.400177+00:00', 'runtime': '811980.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.231', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811984416 ms)
2025-02-17 05:28:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:27:25.400182+00:00', 'runtime': '811920.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.608', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.6', 'ahl_t': '1586.6', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.234', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+811984416 ms)
2025-02-17 05:28:35.595 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811993414 ms)
2025-02-17 05:28:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+811993415 ms)
2025-02-17 05:28:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:28:25.400198+00:00'} (T+811993416 ms)
2025-02-17 05:28:51.835 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812009654 ms)
2025-02-17 05:28:51.835 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812009655 ms)
2025-02-17 05:28:51.836 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58018)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812009655 ms)
2025-02-17 05:29:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812044416 ms)
2025-02-17 05:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812044416 ms)
2025-02-17 05:29:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:29:25.400225+00:00', 'runtime': '812040.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.231', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812044417 ms)
2025-02-17 05:29:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:28:25.400177+00:00', 'runtime': '811980.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.238', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.231', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812044418 ms)
2025-02-17 05:29:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812053417 ms)
2025-02-17 05:29:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812053418 ms)
2025-02-17 05:29:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:29:25.400245+00:00'} (T+812053419 ms)
2025-02-17 05:29:49.998 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812067817 ms)
2025-02-17 05:29:49.998 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812067818 ms)
2025-02-17 05:29:49.999 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58082)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812067818 ms)
2025-02-17 05:30:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812104403 ms)
2025-02-17 05:30:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812104404 ms)
2025-02-17 05:30:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:30:25.400167+00:00', 'runtime': '812100.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.234', 'adc_ic_f': '0.002', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.228', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812104405 ms)
2025-02-17 05:30:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:29:25.400225+00:00', 'runtime': '812040.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.231', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812104405 ms)
2025-02-17 05:30:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812113429 ms)
2025-02-17 05:30:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812113430 ms)
2025-02-17 05:30:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:30:25.400193+00:00'} (T+812113431 ms)
2025-02-17 05:30:40.107 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812117927 ms)
2025-02-17 05:30:40.108 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812117927 ms)
2025-02-17 05:30:40.108 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58136)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812117927 ms)
2025-02-17 05:31:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812164405 ms)
2025-02-17 05:31:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812164405 ms)
2025-02-17 05:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:31:25.400193+00:00', 'runtime': '812160.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.605', 'adc_vl_f': '12.231', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.228', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812164406 ms)
2025-02-17 05:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:30:25.400167+00:00', 'runtime': '812100.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.608', 'adc_vl_f': '12.234', 'adc_ic_f': '0.002', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.228', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812164407 ms)
2025-02-17 05:31:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812173427 ms)
2025-02-17 05:31:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812173428 ms)
2025-02-17 05:31:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:31:25.400203+00:00'} (T+812173429 ms)
2025-02-17 05:31:38.265 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812176085 ms)
2025-02-17 05:31:38.266 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812176085 ms)
2025-02-17 05:31:38.266 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58200)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812176085 ms)
2025-02-17 05:32:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812224411 ms)
2025-02-17 05:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812224411 ms)
2025-02-17 05:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:32:25.400207+00:00', 'runtime': '812220.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.599', 'adc_vl_f': '12.231', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.225', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812224412 ms)
2025-02-17 05:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:31:25.400193+00:00', 'runtime': '812160.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.605', 'adc_vl_f': '12.231', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.228', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812224412 ms)
2025-02-17 05:32:35.859 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812233678 ms)
2025-02-17 05:32:35.859 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812233678 ms)
2025-02-17 05:32:35.860 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:32:25.400217+00:00'} (T+812233679 ms)
2025-02-17 05:33:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812284413 ms)
2025-02-17 05:33:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812284413 ms)
2025-02-17 05:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:33:25.400263+00:00', 'runtime': '812280.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.584', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.225', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812284414 ms)
2025-02-17 05:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:32:25.400207+00:00', 'runtime': '812220.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.599', 'adc_vl_f': '12.231', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.225', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812284415 ms)
2025-02-17 05:33:27.488 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812285307 ms)
2025-02-17 05:33:27.488 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812285307 ms)
2025-02-17 05:33:27.489 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58316)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812285308 ms)
2025-02-17 05:33:35.804 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812293624 ms)
2025-02-17 05:33:35.805 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812293624 ms)
2025-02-17 05:33:35.806 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:33:25.400466+00:00'} (T+812293625 ms)
2025-02-17 05:33:36.577 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812294397 ms)
2025-02-17 05:33:36.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812294397 ms)
2025-02-17 05:33:36.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58328)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812294397 ms)
2025-02-17 05:34:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812344417 ms)
2025-02-17 05:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812344417 ms)
2025-02-17 05:34:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:34:25.400170+00:00', 'runtime': '812340.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.569', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.225', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812344419 ms)
2025-02-17 05:34:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:33:25.400263+00:00', 'runtime': '812280.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.584', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.225', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812344419 ms)
2025-02-17 05:34:29.730 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812347549 ms)
2025-02-17 05:34:29.730 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812347549 ms)
2025-02-17 05:34:29.731 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58384)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812347550 ms)
2025-02-17 05:34:29.770 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812347589 ms)
2025-02-17 05:34:29.770 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812347589 ms)
2025-02-17 05:34:29.771 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58386)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812347590 ms)
2025-02-17 05:34:41.857 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812359676 ms)
2025-02-17 05:34:41.857 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812359676 ms)
2025-02-17 05:34:41.860 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58402)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812359679 ms)
2025-02-17 05:35:25.572 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812403392 ms)
2025-02-17 05:35:25.573 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812403392 ms)
2025-02-17 05:35:25.574 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:35:25.400154+00:00', 'runtime': '812400.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.566', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.222', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812403393 ms)
2025-02-17 05:35:25.574 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:34:25.400170+00:00', 'runtime': '812340.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.569', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.225', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812403394 ms)
2025-02-17 05:35:51.989 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812429808 ms)
2025-02-17 05:35:51.989 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812429808 ms)
2025-02-17 05:35:51.990 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58488)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812429809 ms)
2025-02-17 05:36:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812464407 ms)
2025-02-17 05:36:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812464407 ms)
2025-02-17 05:36:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:36:25.400190+00:00', 'runtime': '812460.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.562', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.222', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812464408 ms)
2025-02-17 05:36:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:35:25.400154+00:00', 'runtime': '812400.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.566', 'adc_vl_f': '12.234', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.241', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.222', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812464409 ms)
2025-02-17 05:36:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812473417 ms)
2025-02-17 05:36:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812473417 ms)
2025-02-17 05:36:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:36:25.400337+00:00'} (T+812473418 ms)
2025-02-17 05:37:14.168 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812511987 ms)
2025-02-17 05:37:14.168 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812511987 ms)
2025-02-17 05:37:14.169 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58576)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812511988 ms)
2025-02-17 05:37:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812524413 ms)
2025-02-17 05:37:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812524413 ms)
2025-02-17 05:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:37:25.400199+00:00', 'runtime': '812520.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.559', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812524414 ms)
2025-02-17 05:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:36:25.400190+00:00', 'runtime': '812460.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.562', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.222', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812524414 ms)
2025-02-17 05:37:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812533430 ms)
2025-02-17 05:37:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812533430 ms)
2025-02-17 05:37:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:37:25.400226+00:00'} (T+812533431 ms)
2025-02-17 05:38:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812584413 ms)
2025-02-17 05:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812584414 ms)
2025-02-17 05:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:38:25.400173+00:00', 'runtime': '812580.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.547', 'adc_vl_f': '12.216', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812584414 ms)
2025-02-17 05:38:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:37:25.400199+00:00', 'runtime': '812520.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.559', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812584415 ms)
2025-02-17 05:38:30.331 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812588150 ms)
2025-02-17 05:38:30.331 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812588150 ms)
2025-02-17 05:38:30.332 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58658)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812588151 ms)
2025-02-17 05:38:35.687 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812593506 ms)
2025-02-17 05:38:35.687 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812593507 ms)
2025-02-17 05:38:35.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:38:25.400198+00:00'} (T+812593508 ms)
2025-02-17 05:39:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812644416 ms)
2025-02-17 05:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812644416 ms)
2025-02-17 05:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:39:25.400172+00:00', 'runtime': '812640.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.535', 'adc_vl_f': '12.228', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812644417 ms)
2025-02-17 05:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:38:25.400173+00:00', 'runtime': '812580.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.547', 'adc_vl_f': '12.216', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812644417 ms)
2025-02-17 05:39:35.500 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812653319 ms)
2025-02-17 05:39:35.500 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812653319 ms)
2025-02-17 05:39:35.500 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58730)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812653320 ms)
2025-02-17 05:40:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812704403 ms)
2025-02-17 05:40:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812704403 ms)
2025-02-17 05:40:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:40:25.400164+00:00', 'runtime': '812700.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.526', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812704404 ms)
2025-02-17 05:40:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:39:25.400172+00:00', 'runtime': '812640.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.535', 'adc_vl_f': '12.228', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812704405 ms)
2025-02-17 05:40:33.652 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812711471 ms)
2025-02-17 05:40:33.652 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812711471 ms)
2025-02-17 05:40:33.652 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58792)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812711471 ms)
2025-02-17 05:41:16.801 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812754620 ms)
2025-02-17 05:41:16.801 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812754620 ms)
2025-02-17 05:41:16.801 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58838)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812754620 ms)
2025-02-17 05:41:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812764406 ms)
2025-02-17 05:41:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812764407 ms)
2025-02-17 05:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:41:25.400267+00:00', 'runtime': '812760.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.523', 'adc_vl_f': '12.222', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812764408 ms)
2025-02-17 05:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:40:25.400164+00:00', 'runtime': '812700.001', 'adc_vb_f': '12.241', 'adc_va_f': '1.526', 'adc_vl_f': '12.241', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.7', 'ahl_t': '1586.7', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812764408 ms)
2025-02-17 05:42:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812824406 ms)
2025-02-17 05:42:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812824407 ms)
2025-02-17 05:42:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:42:25.400181+00:00', 'runtime': '812820.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.52', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812824408 ms)
2025-02-17 05:42:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:41:25.400267+00:00', 'runtime': '812760.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.523', 'adc_vl_f': '12.222', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.219', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812824408 ms)
2025-02-17 05:42:28.942 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812826761 ms)
2025-02-17 05:42:28.942 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812826761 ms)
2025-02-17 05:42:28.943 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58916)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812826762 ms)
2025-02-17 05:43:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812884408 ms)
2025-02-17 05:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812884409 ms)
2025-02-17 05:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:43:25.400176+00:00', 'runtime': '812880.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.511', 'adc_vl_f': '12.216', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812884410 ms)
2025-02-17 05:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:42:25.400181+00:00', 'runtime': '812820.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.52', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812884410 ms)
2025-02-17 05:43:48.122 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+812905942 ms)
2025-02-17 05:43:48.123 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+812905942 ms)
2025-02-17 05:43:48.123 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58998)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+812905942 ms)
2025-02-17 05:44:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+812944406 ms)
2025-02-17 05:44:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+812944407 ms)
2025-02-17 05:44:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:44:25.400187+00:00', 'runtime': '812940.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.21', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812944408 ms)
2025-02-17 05:44:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:43:25.400176+00:00', 'runtime': '812880.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.511', 'adc_vl_f': '12.216', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+812944408 ms)
2025-02-17 05:44:35.762 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812953581 ms)
2025-02-17 05:44:35.762 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+812953582 ms)
2025-02-17 05:44:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:44:25.400345+00:00'} (T+812953582 ms)
2025-02-17 05:45:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813004410 ms)
2025-02-17 05:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813004410 ms)
2025-02-17 05:45:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:45:25.400194+00:00', 'runtime': '813000.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.213', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813004411 ms)
2025-02-17 05:45:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:44:25.400187+00:00', 'runtime': '812940.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.21', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813004414 ms)
2025-02-17 05:46:17.412 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813055231 ms)
2025-02-17 05:46:17.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813055232 ms)
2025-02-17 05:46:17.413 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59168)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813055232 ms)
2025-02-17 05:46:17.453 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813055272 ms)
2025-02-17 05:46:17.453 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813055272 ms)
2025-02-17 05:46:17.453 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59170)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813055273 ms)
2025-02-17 05:46:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813064414 ms)
2025-02-17 05:46:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813064414 ms)
2025-02-17 05:46:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:46:25.400174+00:00', 'runtime': '813060.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.501', 'adc_vl_f': '12.231', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813064415 ms)
2025-02-17 05:46:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:45:25.400194+00:00', 'runtime': '813000.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.213', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813064415 ms)
2025-02-17 05:47:22.632 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813120451 ms)
2025-02-17 05:47:22.632 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813120451 ms)
2025-02-17 05:47:22.632 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59238)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813120452 ms)
2025-02-17 05:47:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813124415 ms)
2025-02-17 05:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813124416 ms)
2025-02-17 05:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:47:25.400170+00:00', 'runtime': '813120.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813124417 ms)
2025-02-17 05:47:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:46:25.400174+00:00', 'runtime': '813060.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.501', 'adc_vl_f': '12.231', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813124417 ms)
2025-02-17 05:48:25.765 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813183584 ms)
2025-02-17 05:48:25.765 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813183584 ms)
2025-02-17 05:48:25.765 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59306)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813183584 ms)
2025-02-17 05:48:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813184405 ms)
2025-02-17 05:48:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813184405 ms)
2025-02-17 05:48:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:48:25.400167+00:00', 'runtime': '813180.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.222', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813184406 ms)
2025-02-17 05:48:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:47:25.400170+00:00', 'runtime': '813120.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.219', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813184407 ms)
2025-02-17 05:49:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813244405 ms)
2025-02-17 05:49:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813244405 ms)
2025-02-17 05:49:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:49:25.400174+00:00', 'runtime': '813240.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.501', 'adc_vl_f': '12.207', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813244406 ms)
2025-02-17 05:49:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:48:25.400167+00:00', 'runtime': '813180.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.222', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813244406 ms)
2025-02-17 05:50:23.021 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813300840 ms)
2025-02-17 05:50:23.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813300841 ms)
2025-02-17 05:50:23.022 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59428)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813300841 ms)
2025-02-17 05:50:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813304407 ms)
2025-02-17 05:50:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813304407 ms)
2025-02-17 05:50:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:50:25.400172+00:00', 'runtime': '813300.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813304408 ms)
2025-02-17 05:50:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:49:25.400174+00:00', 'runtime': '813240.001', 'adc_vb_f': '12.238', 'adc_va_f': '1.501', 'adc_vl_f': '12.207', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813304408 ms)
2025-02-17 05:51:23.173 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813360992 ms)
2025-02-17 05:51:23.173 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813360992 ms)
2025-02-17 05:51:23.173 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59494)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813360993 ms)
2025-02-17 05:51:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813364406 ms)
2025-02-17 05:51:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813364406 ms)
2025-02-17 05:51:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:51:25.400758+00:00', 'runtime': '813360.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.207', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813364407 ms)
2025-02-17 05:51:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:50:25.400172+00:00', 'runtime': '813300.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813364408 ms)
2025-02-17 05:51:50.275 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813388094 ms)
2025-02-17 05:51:50.275 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813388095 ms)
2025-02-17 05:51:50.276 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59524)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813388095 ms)
2025-02-17 05:52:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813424410 ms)
2025-02-17 05:52:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813424410 ms)
2025-02-17 05:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:52:25.400597+00:00', 'runtime': '813420.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.219', 'adc_ic_f': '0.002', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813424411 ms)
2025-02-17 05:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:51:25.400758+00:00', 'runtime': '813360.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.207', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813424412 ms)
2025-02-17 05:53:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813484412 ms)
2025-02-17 05:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813484412 ms)
2025-02-17 05:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:53:25.402767+00:00', 'runtime': '813480.003', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.213', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813484413 ms)
2025-02-17 05:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:52:25.400597+00:00', 'runtime': '813420.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.219', 'adc_ic_f': '0.002', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.234', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813484413 ms)
2025-02-17 05:54:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813544413 ms)
2025-02-17 05:54:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813544414 ms)
2025-02-17 05:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:54:25.400185+00:00', 'runtime': '813540.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.207', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813544415 ms)
2025-02-17 05:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:53:25.402767+00:00', 'runtime': '813480.003', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.213', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.8', 'ahl_t': '1586.8', 'hourmeter': '20112', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813544415 ms)
2025-02-17 05:54:38.536 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813556355 ms)
2025-02-17 05:54:38.536 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813556355 ms)
2025-02-17 05:54:38.537 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59700)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813556356 ms)
2025-02-17 05:54:57.626 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813575445 ms)
2025-02-17 05:54:57.626 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813575446 ms)
2025-02-17 05:54:57.627 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59722)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813575446 ms)
2025-02-17 05:55:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813604417 ms)
2025-02-17 05:55:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813604418 ms)
2025-02-17 05:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:55:25.400258+00:00', 'runtime': '813600.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813604419 ms)
2025-02-17 05:55:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:54:25.400185+00:00', 'runtime': '813540.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.501', 'adc_vl_f': '12.207', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813604419 ms)
2025-02-17 05:55:35.864 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+813613683 ms)
2025-02-17 05:55:35.865 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+813613684 ms)
2025-02-17 05:55:35.866 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:55:25.400352+00:00'} (T+813613685 ms)
2025-02-17 05:55:50.774 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813628593 ms)
2025-02-17 05:55:50.774 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813628593 ms)
2025-02-17 05:55:50.774 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59790)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813628594 ms)
2025-02-17 05:55:59.876 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813637695 ms)
2025-02-17 05:55:59.876 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813637695 ms)
2025-02-17 05:55:59.876 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59800)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813637696 ms)
2025-02-17 05:56:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813664402 ms)
2025-02-17 05:56:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813664403 ms)
2025-02-17 05:56:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:56:25.400160+00:00', 'runtime': '813660.001', 'adc_vb_f': '12.228', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813664404 ms)
2025-02-17 05:56:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:55:25.400258+00:00', 'runtime': '813600.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.231', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813664404 ms)
2025-02-17 05:56:33.979 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813671798 ms)
2025-02-17 05:56:33.979 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813671798 ms)
2025-02-17 05:56:33.980 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59838)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813671799 ms)
2025-02-17 05:56:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+813673427 ms)
2025-02-17 05:56:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+813673428 ms)
2025-02-17 05:56:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:56:25.400429+00:00'} (T+813673429 ms)
2025-02-17 05:57:25.137 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813722956 ms)
2025-02-17 05:57:25.137 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813722957 ms)
2025-02-17 05:57:25.138 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59892)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813722957 ms)
2025-02-17 05:57:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813724404 ms)
2025-02-17 05:57:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813724405 ms)
2025-02-17 05:57:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:57:25.400236+00:00', 'runtime': '813720.001', 'adc_vb_f': '12.228', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813724408 ms)
2025-02-17 05:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:56:25.400160+00:00', 'runtime': '813660.001', 'adc_vb_f': '12.228', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813724409 ms)
2025-02-17 05:57:35.446 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+813733265 ms)
2025-02-17 05:57:35.447 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+813733266 ms)
2025-02-17 05:57:35.448 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 05:57:25.400244+00:00'} (T+813733267 ms)
2025-02-17 05:57:49.225 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813747044 ms)
2025-02-17 05:57:49.225 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813747044 ms)
2025-02-17 05:57:49.226 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59922)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813747045 ms)
2025-02-17 05:58:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813784420 ms)
2025-02-17 05:58:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813784421 ms)
2025-02-17 05:58:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:58:25.400187+00:00', 'runtime': '813780.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813784423 ms)
2025-02-17 05:58:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:57:25.400236+00:00', 'runtime': '813720.001', 'adc_vb_f': '12.228', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813784423 ms)
2025-02-17 05:59:26.427 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+813844246 ms)
2025-02-17 05:59:26.427 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+813844247 ms)
2025-02-17 05:59:26.428 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60024)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+813844247 ms)
2025-02-17 05:59:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813844406 ms)
2025-02-17 05:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813844407 ms)
2025-02-17 05:59:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 05:59:25.400073+00:00', 'runtime': '813840.001', 'adc_vb_f': '12.228', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813844409 ms)
2025-02-17 05:59:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:58:25.400187+00:00', 'runtime': '813780.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813844409 ms)
2025-02-17 06:00:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813904409 ms)
2025-02-17 06:00:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813904409 ms)
2025-02-17 06:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:00:25.400760+00:00', 'runtime': '813900.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813904413 ms)
2025-02-17 06:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 05:59:25.400073+00:00', 'runtime': '813840.001', 'adc_vb_f': '12.228', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813904413 ms)
2025-02-17 06:01:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+813964409 ms)
2025-02-17 06:01:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+813964410 ms)
2025-02-17 06:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:01:25.402797+00:00', 'runtime': '813960.003', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813964412 ms)
2025-02-17 06:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:00:25.400760+00:00', 'runtime': '813900.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+813964412 ms)
2025-02-17 06:02:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814024411 ms)
2025-02-17 06:02:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814024411 ms)
2025-02-17 06:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:02:25.400817+00:00', 'runtime': '814020.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814024413 ms)
2025-02-17 06:02:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:01:25.402797+00:00', 'runtime': '813960.003', 'adc_vb_f': '12.231', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814024414 ms)
2025-02-17 06:02:52.728 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814050547 ms)
2025-02-17 06:02:52.728 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814050548 ms)
2025-02-17 06:02:52.729 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60242)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814050548 ms)
2025-02-17 06:03:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814084413 ms)
2025-02-17 06:03:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814084413 ms)
2025-02-17 06:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:03:25.400600+00:00', 'runtime': '814080.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.225', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814084415 ms)
2025-02-17 06:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:02:25.400817+00:00', 'runtime': '814020.001', 'adc_vb_f': '12.231', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.228', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814084416 ms)
2025-02-17 06:04:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814144414 ms)
2025-02-17 06:04:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814144414 ms)
2025-02-17 06:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:04:25.400193+00:00', 'runtime': '814140.001', 'adc_vb_f': '12.225', 'adc_va_f': '1.486', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.225', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814144415 ms)
2025-02-17 06:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:03:25.400600+00:00', 'runtime': '814080.001', 'adc_vb_f': '12.234', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.225', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814144416 ms)
2025-02-17 06:04:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814153438 ms)
2025-02-17 06:04:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814153438 ms)
2025-02-17 06:04:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:04:25.413049+00:00'} (T+814153440 ms)
2025-02-17 06:05:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814204416 ms)
2025-02-17 06:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814204416 ms)
2025-02-17 06:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:05:25.402822+00:00', 'runtime': '814200.003', 'adc_vb_f': '12.222', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.225', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814204417 ms)
2025-02-17 06:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:04:25.400193+00:00', 'runtime': '814140.001', 'adc_vb_f': '12.225', 'adc_va_f': '1.486', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.225', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814204418 ms)
2025-02-17 06:05:59.038 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814236858 ms)
2025-02-17 06:05:59.039 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814236858 ms)
2025-02-17 06:05:59.039 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60450)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814236858 ms)
2025-02-17 06:06:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814264415 ms)
2025-02-17 06:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814264415 ms)
2025-02-17 06:06:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:06:25.400846+00:00', 'runtime': '814260.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.486', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814264418 ms)
2025-02-17 06:06:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:05:25.402822+00:00', 'runtime': '814200.003', 'adc_vb_f': '12.222', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.225', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814264419 ms)
2025-02-17 06:07:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814324416 ms)
2025-02-17 06:07:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814324417 ms)
2025-02-17 06:07:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:07:25.400589+00:00', 'runtime': '814320.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814324417 ms)
2025-02-17 06:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:06:25.400846+00:00', 'runtime': '814260.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.486', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1586.9', 'ahl_t': '1586.9', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814324418 ms)
2025-02-17 06:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814384414 ms)
2025-02-17 06:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814384415 ms)
2025-02-17 06:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:08:25.400184+00:00', 'runtime': '814380.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814384416 ms)
2025-02-17 06:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:07:25.400589+00:00', 'runtime': '814320.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814384416 ms)
2025-02-17 06:08:35.676 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814393495 ms)
2025-02-17 06:08:35.677 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814393496 ms)
2025-02-17 06:08:35.678 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:08:25.401082+00:00'} (T+814393497 ms)
2025-02-17 06:08:56.343 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814414162 ms)
2025-02-17 06:08:56.343 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814414163 ms)
2025-02-17 06:08:56.344 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60636)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814414163 ms)
2025-02-17 06:09:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814444413 ms)
2025-02-17 06:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814444414 ms)
2025-02-17 06:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:09:25.402846+00:00', 'runtime': '814440.003', 'adc_vb_f': '12.222', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814444416 ms)
2025-02-17 06:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:08:25.400184+00:00', 'runtime': '814380.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814444416 ms)
2025-02-17 06:09:28.384 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814446203 ms)
2025-02-17 06:09:28.384 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814446204 ms)
2025-02-17 06:09:28.385 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814446204 ms)
2025-02-17 06:09:28.386 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814446205 ms)
2025-02-17 06:09:28.386 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814446205 ms)
2025-02-17 06:09:28.386 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814446206 ms)
2025-02-17 06:09:28.387 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814446206 ms)
2025-02-17 06:09:28.387 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814446207 ms)
2025-02-17 06:09:28.388 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814446207 ms)
2025-02-17 06:09:28.389 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814446208 ms)
2025-02-17 06:09:28.389 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814446208 ms)
2025-02-17 06:09:28.389 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814446209 ms)
2025-02-17 06:09:28.390 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814446209 ms)
2025-02-17 06:09:28.390 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814446209 ms)
2025-02-17 06:09:28.391 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814446210 ms)
2025-02-17 06:09:28.391 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814446211 ms)
2025-02-17 06:09:28.392 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814446211 ms)
2025-02-17 06:09:28.392 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814446211 ms)
2025-02-17 06:09:28.393 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814446212 ms)
2025-02-17 06:09:28.394 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814446213 ms)
2025-02-17 06:09:28.411 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814446231 ms)
2025-02-17 06:09:35.708 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814453527 ms)
2025-02-17 06:09:35.709 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814453528 ms)
2025-02-17 06:09:35.710 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:09:25.400116+00:00'} (T+814453529 ms)
2025-02-17 06:09:38.599 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814456419 ms)
2025-02-17 06:09:38.600 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814456419 ms)
2025-02-17 06:09:38.600 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60682)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814456419 ms)
2025-02-17 06:10:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814504413 ms)
2025-02-17 06:10:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814504414 ms)
2025-02-17 06:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:10:25.400099+00:00', 'runtime': '814500.001', 'adc_vb_f': '12.222', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814504418 ms)
2025-02-17 06:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:09:25.402846+00:00', 'runtime': '814440.003', 'adc_vb_f': '12.222', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814504418 ms)
2025-02-17 06:10:35.622 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814513441 ms)
2025-02-17 06:10:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814513441 ms)
2025-02-17 06:10:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:10:25.413123+00:00'} (T+814513443 ms)
2025-02-17 06:11:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814564413 ms)
2025-02-17 06:11:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814564413 ms)
2025-02-17 06:11:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:11:25.402770+00:00', 'runtime': '814560.003', 'adc_vb_f': '12.222', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814564417 ms)
2025-02-17 06:11:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:10:25.400099+00:00', 'runtime': '814500.001', 'adc_vb_f': '12.222', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.222', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814564417 ms)
2025-02-17 06:11:52.802 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814590622 ms)
2025-02-17 06:11:52.803 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814590622 ms)
2025-02-17 06:11:52.803 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60826)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814590622 ms)
2025-02-17 06:12:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814624412 ms)
2025-02-17 06:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814624413 ms)
2025-02-17 06:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:12:25.400179+00:00', 'runtime': '814620.001', 'adc_vb_f': '12.222', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814624415 ms)
2025-02-17 06:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:11:25.402770+00:00', 'runtime': '814560.003', 'adc_vb_f': '12.222', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814624415 ms)
2025-02-17 06:13:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814684412 ms)
2025-02-17 06:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814684412 ms)
2025-02-17 06:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:13:25.400173+00:00', 'runtime': '814680.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814684414 ms)
2025-02-17 06:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:12:25.400179+00:00', 'runtime': '814620.001', 'adc_vb_f': '12.222', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814684415 ms)
2025-02-17 06:13:36.154 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814693974 ms)
2025-02-17 06:13:36.155 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814693974 ms)
2025-02-17 06:13:36.159 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:13:25.400321+00:00'} (T+814693979 ms)
2025-02-17 06:13:48.032 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814705851 ms)
2025-02-17 06:13:48.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814705851 ms)
2025-02-17 06:13:48.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60946)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814705852 ms)
2025-02-17 06:14:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814744412 ms)
2025-02-17 06:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814744412 ms)
2025-02-17 06:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:14:25.402738+00:00', 'runtime': '814740.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.44', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814744413 ms)
2025-02-17 06:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:13:25.400173+00:00', 'runtime': '814680.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814744413 ms)
2025-02-17 06:14:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814753431 ms)
2025-02-17 06:14:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814753432 ms)
2025-02-17 06:14:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:14:25.400130+00:00'} (T+814753434 ms)
2025-02-17 06:14:40.182 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814758001 ms)
2025-02-17 06:14:40.182 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814758001 ms)
2025-02-17 06:14:40.182 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 32772)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814758002 ms)
2025-02-17 06:15:18.300 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814796119 ms)
2025-02-17 06:15:18.300 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814796119 ms)
2025-02-17 06:15:18.300 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 32814)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814796119 ms)
2025-02-17 06:15:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814804412 ms)
2025-02-17 06:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814804412 ms)
2025-02-17 06:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:15:25.400210+00:00', 'runtime': '814800.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814804416 ms)
2025-02-17 06:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:14:25.402738+00:00', 'runtime': '814740.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.44', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814804416 ms)
2025-02-17 06:15:35.807 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814813627 ms)
2025-02-17 06:15:35.808 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814813627 ms)
2025-02-17 06:15:35.812 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:15:25.400215+00:00'} (T+814813632 ms)
2025-02-17 06:15:57.447 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814835266 ms)
2025-02-17 06:15:57.447 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814835267 ms)
2025-02-17 06:15:57.448 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 32868)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814835267 ms)
2025-02-17 06:16:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814864412 ms)
2025-02-17 06:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814864413 ms)
2025-02-17 06:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:16:25.400179+00:00', 'runtime': '814860.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814864414 ms)
2025-02-17 06:16:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:15:25.400210+00:00', 'runtime': '814800.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814864415 ms)
2025-02-17 06:16:35.653 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814873473 ms)
2025-02-17 06:16:35.654 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814873473 ms)
2025-02-17 06:16:35.656 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:16:25.399995+00:00'} (T+814873475 ms)
2025-02-17 06:17:05.580 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+814903399 ms)
2025-02-17 06:17:05.580 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+814903399 ms)
2025-02-17 06:17:05.581 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 32942)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814903400 ms)
2025-02-17 06:17:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814924413 ms)
2025-02-17 06:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814924413 ms)
2025-02-17 06:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:17:25.400165+00:00', 'runtime': '814920.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814924415 ms)
2025-02-17 06:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:16:25.400179+00:00', 'runtime': '814860.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814924416 ms)
2025-02-17 06:17:35.820 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814933640 ms)
2025-02-17 06:17:35.821 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814933640 ms)
2025-02-17 06:17:35.826 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:17:25.400131+00:00'} (T+814933645 ms)
2025-02-17 06:18:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+814984413 ms)
2025-02-17 06:18:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+814984414 ms)
2025-02-17 06:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:18:25.400543+00:00', 'runtime': '814980.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.428', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814984418 ms)
2025-02-17 06:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:17:25.400165+00:00', 'runtime': '814920.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+814984418 ms)
2025-02-17 06:18:29.684 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+814987503 ms)
2025-02-17 06:18:29.685 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+814987505 ms)
2025-02-17 06:18:29.689 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+814987508 ms)
2025-02-17 06:18:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814993427 ms)
2025-02-17 06:18:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+814993428 ms)
2025-02-17 06:18:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:18:25.399995+00:00'} (T+814993430 ms)
2025-02-17 06:19:15.818 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815033637 ms)
2025-02-17 06:19:15.818 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815033637 ms)
2025-02-17 06:19:15.818 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33082)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815033638 ms)
2025-02-17 06:19:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815044415 ms)
2025-02-17 06:19:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815044415 ms)
2025-02-17 06:19:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:19:25.400206+00:00', 'runtime': '815040.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.45', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815044419 ms)
2025-02-17 06:19:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:18:25.400543+00:00', 'runtime': '814980.001', 'adc_vb_f': '12.219', 'adc_va_f': '1.428', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.0', 'ahl_t': '1587.0', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815044419 ms)
2025-02-17 06:19:35.881 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815053700 ms)
2025-02-17 06:19:35.881 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815053701 ms)
2025-02-17 06:19:35.886 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:19:25.400227+00:00'} (T+815053705 ms)
2025-02-17 06:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815104412 ms)
2025-02-17 06:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815104412 ms)
2025-02-17 06:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:20:25.400165+00:00', 'runtime': '815100.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815104413 ms)
2025-02-17 06:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:19:25.400206+00:00', 'runtime': '815040.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.45', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815104414 ms)
2025-02-17 06:20:35.648 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815113468 ms)
2025-02-17 06:20:35.649 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815113468 ms)
2025-02-17 06:20:35.650 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:20:25.400404+00:00'} (T+815113470 ms)
2025-02-17 06:21:26.055 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815163874 ms)
2025-02-17 06:21:26.055 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815163874 ms)
2025-02-17 06:21:26.055 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33220)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815163875 ms)
2025-02-17 06:21:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815164413 ms)
2025-02-17 06:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815164414 ms)
2025-02-17 06:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:21:25.400188+00:00', 'runtime': '815160.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815164416 ms)
2025-02-17 06:21:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:20:25.400165+00:00', 'runtime': '815100.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815164417 ms)
2025-02-17 06:21:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815173425 ms)
2025-02-17 06:21:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815173426 ms)
2025-02-17 06:21:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:21:25.400154+00:00'} (T+815173432 ms)
2025-02-17 06:22:21.194 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815219013 ms)
2025-02-17 06:22:21.194 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815219013 ms)
2025-02-17 06:22:21.194 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33280)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815219013 ms)
2025-02-17 06:22:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815224411 ms)
2025-02-17 06:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815224412 ms)
2025-02-17 06:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:22:25.402749+00:00', 'runtime': '815220.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815224413 ms)
2025-02-17 06:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:21:25.400188+00:00', 'runtime': '815160.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815224413 ms)
2025-02-17 06:22:35.672 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815233491 ms)
2025-02-17 06:22:35.672 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815233491 ms)
2025-02-17 06:22:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:22:25.400177+00:00'} (T+815233493 ms)
2025-02-17 06:23:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815284412 ms)
2025-02-17 06:23:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815284413 ms)
2025-02-17 06:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:23:25.402759+00:00', 'runtime': '815280.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815284414 ms)
2025-02-17 06:23:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:22:25.402749+00:00', 'runtime': '815220.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815284415 ms)
2025-02-17 06:23:35.758 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815293577 ms)
2025-02-17 06:23:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815293578 ms)
2025-02-17 06:23:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:23:25.400139+00:00'} (T+815293578 ms)
2025-02-17 06:24:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815344413 ms)
2025-02-17 06:24:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815344413 ms)
2025-02-17 06:24:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:24:25.402659+00:00', 'runtime': '815340.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815344417 ms)
2025-02-17 06:24:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:23:25.402759+00:00', 'runtime': '815280.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815344418 ms)
2025-02-17 06:24:35.650 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815353470 ms)
2025-02-17 06:24:35.651 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815353470 ms)
2025-02-17 06:24:35.652 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:24:25.400050+00:00'} (T+815353471 ms)
2025-02-17 06:25:14.492 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815392311 ms)
2025-02-17 06:25:14.492 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815392312 ms)
2025-02-17 06:25:14.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33466)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815392312 ms)
2025-02-17 06:25:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815404416 ms)
2025-02-17 06:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815404416 ms)
2025-02-17 06:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:25:25.400141+00:00', 'runtime': '815400.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.453', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815404420 ms)
2025-02-17 06:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:24:25.402659+00:00', 'runtime': '815340.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815404420 ms)
2025-02-17 06:25:33.572 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815411391 ms)
2025-02-17 06:25:33.572 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815411391 ms)
2025-02-17 06:25:33.572 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33486)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815411392 ms)
2025-02-17 06:25:35.822 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815413641 ms)
2025-02-17 06:25:35.822 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815413642 ms)
2025-02-17 06:25:35.823 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:25:25.413126+00:00'} (T+815413643 ms)
2025-02-17 06:26:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815464417 ms)
2025-02-17 06:26:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815464417 ms)
2025-02-17 06:26:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:26:25.400021+00:00', 'runtime': '815460.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815464418 ms)
2025-02-17 06:26:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:25:25.400141+00:00', 'runtime': '815400.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.453', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815464419 ms)
2025-02-17 06:26:35.656 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815473476 ms)
2025-02-17 06:26:35.657 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815473476 ms)
2025-02-17 06:26:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:26:25.409123+00:00'} (T+815473477 ms)
2025-02-17 06:27:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815524420 ms)
2025-02-17 06:27:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815524420 ms)
2025-02-17 06:27:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:27:25.402738+00:00', 'runtime': '815520.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815524421 ms)
2025-02-17 06:27:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:26:25.400021+00:00', 'runtime': '815460.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815524422 ms)
2025-02-17 06:27:35.626 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815533446 ms)
2025-02-17 06:27:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815533446 ms)
2025-02-17 06:27:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:27:25.400147+00:00'} (T+815533447 ms)
2025-02-17 06:27:37.731 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+815535551 ms)
2025-02-17 06:27:37.732 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+815535551 ms)
2025-02-17 06:27:37.732 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815535551 ms)
2025-02-17 06:27:37.733 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+815535552 ms)
2025-02-17 06:27:37.733 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+815535552 ms)
2025-02-17 06:27:37.733 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815535552 ms)
2025-02-17 06:27:37.734 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+815535553 ms)
2025-02-17 06:27:37.734 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+815535553 ms)
2025-02-17 06:27:37.734 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815535553 ms)
2025-02-17 06:27:37.735 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+815535554 ms)
2025-02-17 06:27:37.735 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+815535554 ms)
2025-02-17 06:27:37.735 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815535554 ms)
2025-02-17 06:27:37.735 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+815535555 ms)
2025-02-17 06:27:37.736 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+815535555 ms)
2025-02-17 06:27:37.736 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815535555 ms)
2025-02-17 06:27:37.736 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+815535556 ms)
2025-02-17 06:27:37.737 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+815535556 ms)
2025-02-17 06:27:37.737 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815535556 ms)
2025-02-17 06:28:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815584407 ms)
2025-02-17 06:28:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815584407 ms)
2025-02-17 06:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:28:25.402737+00:00', 'runtime': '815580.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815584408 ms)
2025-02-17 06:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:27:25.402738+00:00', 'runtime': '815520.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815584409 ms)
2025-02-17 06:28:35.629 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815593448 ms)
2025-02-17 06:28:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815593449 ms)
2025-02-17 06:28:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:28:25.400171+00:00'} (T+815593450 ms)
2025-02-17 06:29:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815644422 ms)
2025-02-17 06:29:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815644422 ms)
2025-02-17 06:29:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:29:25.402758+00:00', 'runtime': '815640.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815644423 ms)
2025-02-17 06:29:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:28:25.402737+00:00', 'runtime': '815580.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815644424 ms)
2025-02-17 06:29:56.981 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815674801 ms)
2025-02-17 06:29:56.982 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815674801 ms)
2025-02-17 06:29:56.982 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33780)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815674801 ms)
2025-02-17 06:30:13.081 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815690900 ms)
2025-02-17 06:30:13.081 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815690900 ms)
2025-02-17 06:30:13.082 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33798)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815690901 ms)
2025-02-17 06:30:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815704407 ms)
2025-02-17 06:30:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815704407 ms)
2025-02-17 06:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:30:25.400182+00:00', 'runtime': '815700.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815704408 ms)
2025-02-17 06:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:29:25.402758+00:00', 'runtime': '815640.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815704409 ms)
2025-02-17 06:31:14.232 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815752051 ms)
2025-02-17 06:31:14.232 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815752052 ms)
2025-02-17 06:31:14.233 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33860)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815752052 ms)
2025-02-17 06:31:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815764408 ms)
2025-02-17 06:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815764408 ms)
2025-02-17 06:31:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:31:25.400185+00:00', 'runtime': '815760.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815764409 ms)
2025-02-17 06:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:30:25.400182+00:00', 'runtime': '815700.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815764410 ms)
2025-02-17 06:31:36.626 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815774445 ms)
2025-02-17 06:31:36.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815774445 ms)
2025-02-17 06:31:36.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:31:25.400332+00:00'} (T+815774446 ms)
2025-02-17 06:31:46.349 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815784168 ms)
2025-02-17 06:31:46.349 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815784168 ms)
2025-02-17 06:31:46.349 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33896)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815784169 ms)
2025-02-17 06:32:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815824409 ms)
2025-02-17 06:32:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815824410 ms)
2025-02-17 06:32:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:32:25.400218+00:00', 'runtime': '815820.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815824411 ms)
2025-02-17 06:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:31:25.400185+00:00', 'runtime': '815760.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.1', 'ahl_t': '1587.1', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815824411 ms)
2025-02-17 06:32:35.649 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815833468 ms)
2025-02-17 06:32:35.649 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815833468 ms)
2025-02-17 06:32:35.650 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:32:25.400211+00:00'} (T+815833469 ms)
2025-02-17 06:33:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815884410 ms)
2025-02-17 06:33:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815884411 ms)
2025-02-17 06:33:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:33:25.400203+00:00', 'runtime': '815880.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815884412 ms)
2025-02-17 06:33:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:32:25.400218+00:00', 'runtime': '815820.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815884412 ms)
2025-02-17 06:33:35.818 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815893637 ms)
2025-02-17 06:33:35.818 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815893637 ms)
2025-02-17 06:33:35.819 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:33:25.400187+00:00'} (T+815893638 ms)
2025-02-17 06:34:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+815944411 ms)
2025-02-17 06:34:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+815944412 ms)
2025-02-17 06:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:34:25.400181+00:00', 'runtime': '815940.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815944413 ms)
2025-02-17 06:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:33:25.400203+00:00', 'runtime': '815880.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+815944413 ms)
2025-02-17 06:34:26.601 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815944420 ms)
2025-02-17 06:34:26.601 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815944421 ms)
2025-02-17 06:34:26.602 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34068)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815944421 ms)
2025-02-17 06:34:35.908 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815953728 ms)
2025-02-17 06:34:35.909 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+815953728 ms)
2025-02-17 06:34:35.910 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:34:25.400006+00:00'} (T+815953729 ms)
2025-02-17 06:34:55.738 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+815973557 ms)
2025-02-17 06:34:55.738 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+815973557 ms)
2025-02-17 06:34:55.739 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34102)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+815973558 ms)
2025-02-17 06:35:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816004413 ms)
2025-02-17 06:35:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816004413 ms)
2025-02-17 06:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:35:25.402790+00:00', 'runtime': '816000.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816004414 ms)
2025-02-17 06:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:34:25.400181+00:00', 'runtime': '815940.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816004415 ms)
2025-02-17 06:35:35.886 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816013705 ms)
2025-02-17 06:35:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816013706 ms)
2025-02-17 06:35:35.888 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:35:25.400206+00:00'} (T+816013707 ms)
2025-02-17 06:36:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816064416 ms)
2025-02-17 06:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816064416 ms)
2025-02-17 06:36:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:36:25.402811+00:00', 'runtime': '816060.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816064417 ms)
2025-02-17 06:36:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:35:25.402790+00:00', 'runtime': '816000.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816064418 ms)
2025-02-17 06:36:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816073418 ms)
2025-02-17 06:36:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816073419 ms)
2025-02-17 06:36:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:36:25.400204+00:00'} (T+816073420 ms)
2025-02-17 06:37:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816124418 ms)
2025-02-17 06:37:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816124419 ms)
2025-02-17 06:37:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:37:25.400174+00:00', 'runtime': '816120.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816124419 ms)
2025-02-17 06:37:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:36:25.402811+00:00', 'runtime': '816060.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816124420 ms)
2025-02-17 06:37:35.645 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816133464 ms)
2025-02-17 06:37:35.646 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816133465 ms)
2025-02-17 06:37:35.647 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:37:25.413123+00:00'} (T+816133466 ms)
2025-02-17 06:38:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816184418 ms)
2025-02-17 06:38:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816184419 ms)
2025-02-17 06:38:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:38:25.402793+00:00', 'runtime': '816180.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.507', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816184420 ms)
2025-02-17 06:38:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:37:25.400174+00:00', 'runtime': '816120.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816184420 ms)
2025-02-17 06:38:27.067 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816184886 ms)
2025-02-17 06:38:27.067 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816184886 ms)
2025-02-17 06:38:27.067 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34336)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816184886 ms)
2025-02-17 06:38:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816193438 ms)
2025-02-17 06:38:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816193438 ms)
2025-02-17 06:38:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:38:25.400141+00:00'} (T+816193439 ms)
2025-02-17 06:39:19.196 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816237015 ms)
2025-02-17 06:39:19.196 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816237016 ms)
2025-02-17 06:39:19.197 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34394)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816237016 ms)
2025-02-17 06:39:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816244417 ms)
2025-02-17 06:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816244417 ms)
2025-02-17 06:39:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:39:25.402763+00:00', 'runtime': '816240.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.51', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816244418 ms)
2025-02-17 06:39:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:38:25.402793+00:00', 'runtime': '816180.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.507', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816244419 ms)
2025-02-17 06:39:32.281 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816250100 ms)
2025-02-17 06:39:32.281 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816250101 ms)
2025-02-17 06:39:32.282 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34408)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816250101 ms)
2025-02-17 06:39:35.775 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816253595 ms)
2025-02-17 06:39:35.776 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816253595 ms)
2025-02-17 06:39:35.777 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:39:25.400175+00:00'} (T+816253596 ms)
2025-02-17 06:39:56.387 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816274206 ms)
2025-02-17 06:39:56.387 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816274206 ms)
2025-02-17 06:39:56.388 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34438)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816274207 ms)
2025-02-17 06:40:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816304421 ms)
2025-02-17 06:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816304421 ms)
2025-02-17 06:40:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:40:25.401108+00:00', 'runtime': '816300.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816304422 ms)
2025-02-17 06:40:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:39:25.402763+00:00', 'runtime': '816240.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.51', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816304422 ms)
2025-02-17 06:40:35.655 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816313474 ms)
2025-02-17 06:40:35.655 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816313475 ms)
2025-02-17 06:40:35.656 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:40:25.400009+00:00'} (T+816313475 ms)
2025-02-17 06:41:24.569 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816362388 ms)
2025-02-17 06:41:24.569 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816362388 ms)
2025-02-17 06:41:24.570 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34532)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816362389 ms)
2025-02-17 06:41:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816364420 ms)
2025-02-17 06:41:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816364421 ms)
2025-02-17 06:41:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:41:25.400127+00:00', 'runtime': '816360.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816364422 ms)
2025-02-17 06:41:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:40:25.401108+00:00', 'runtime': '816300.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816364422 ms)
2025-02-17 06:41:35.816 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816373635 ms)
2025-02-17 06:41:35.816 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816373636 ms)
2025-02-17 06:41:35.817 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:41:25.413133+00:00'} (T+816373637 ms)
2025-02-17 06:42:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816424421 ms)
2025-02-17 06:42:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816424422 ms)
2025-02-17 06:42:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:42:25.400003+00:00', 'runtime': '816420.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816424423 ms)
2025-02-17 06:42:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:41:25.400127+00:00', 'runtime': '816360.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816424423 ms)
2025-02-17 06:42:35.696 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816433515 ms)
2025-02-17 06:42:35.696 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816433515 ms)
2025-02-17 06:42:35.697 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:42:25.409117+00:00'} (T+816433517 ms)
2025-02-17 06:43:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816484409 ms)
2025-02-17 06:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816484409 ms)
2025-02-17 06:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:43:25.401126+00:00', 'runtime': '816480.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816484410 ms)
2025-02-17 06:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:42:25.400003+00:00', 'runtime': '816420.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816484410 ms)
2025-02-17 06:43:36.080 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816493899 ms)
2025-02-17 06:43:36.080 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816493899 ms)
2025-02-17 06:43:36.081 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:43:25.400113+00:00'} (T+816493901 ms)
2025-02-17 06:44:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816544408 ms)
2025-02-17 06:44:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816544409 ms)
2025-02-17 06:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:44:25.400184+00:00', 'runtime': '816540.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816544410 ms)
2025-02-17 06:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:43:25.401126+00:00', 'runtime': '816480.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816544410 ms)
2025-02-17 06:45:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816604412 ms)
2025-02-17 06:45:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816604413 ms)
2025-02-17 06:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:45:25.400802+00:00', 'runtime': '816600.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816604414 ms)
2025-02-17 06:45:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:44:25.400184+00:00', 'runtime': '816540.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.2', 'ahl_t': '1587.2', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816604414 ms)
2025-02-17 06:45:28.872 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+816606692 ms)
2025-02-17 06:45:28.873 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+816606692 ms)
2025-02-17 06:45:28.873 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816606693 ms)
2025-02-17 06:45:28.874 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+816606693 ms)
2025-02-17 06:45:28.874 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+816606693 ms)
2025-02-17 06:45:28.875 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816606694 ms)
2025-02-17 06:45:28.875 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+816606694 ms)
2025-02-17 06:45:28.875 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+816606695 ms)
2025-02-17 06:45:28.876 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816606695 ms)
2025-02-17 06:45:28.876 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+816606695 ms)
2025-02-17 06:45:28.876 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+816606696 ms)
2025-02-17 06:45:28.877 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816606696 ms)
2025-02-17 06:45:28.878 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+816606697 ms)
2025-02-17 06:45:28.878 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+816606697 ms)
2025-02-17 06:45:28.878 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816606697 ms)
2025-02-17 06:45:28.879 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+816606698 ms)
2025-02-17 06:45:28.879 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+816606698 ms)
2025-02-17 06:45:28.880 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816606699 ms)
2025-02-17 06:45:28.880 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+816606700 ms)
2025-02-17 06:45:28.881 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+816606700 ms)
2025-02-17 06:45:28.881 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816606701 ms)
2025-02-17 06:46:34.490 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816672309 ms)
2025-02-17 06:46:34.491 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816672310 ms)
2025-02-17 06:46:34.492 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:46:25.400177+00:00', 'runtime': '816660.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816672311 ms)
2025-02-17 06:46:34.492 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:45:25.400802+00:00', 'runtime': '816600.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816672311 ms)
2025-02-17 06:47:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816724418 ms)
2025-02-17 06:47:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816724418 ms)
2025-02-17 06:47:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:47:25.400303+00:00', 'runtime': '816720.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816724419 ms)
2025-02-17 06:47:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:46:25.400177+00:00', 'runtime': '816660.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816724420 ms)
2025-02-17 06:47:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816733437 ms)
2025-02-17 06:47:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816733437 ms)
2025-02-17 06:47:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:47:25.413051+00:00'} (T+816733438 ms)
2025-02-17 06:47:50.126 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816747946 ms)
2025-02-17 06:47:50.127 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816747946 ms)
2025-02-17 06:47:50.127 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34952)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816747946 ms)
2025-02-17 06:47:54.209 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816752029 ms)
2025-02-17 06:47:54.210 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816752029 ms)
2025-02-17 06:47:54.210 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34958)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816752029 ms)
2025-02-17 06:48:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816784418 ms)
2025-02-17 06:48:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816784418 ms)
2025-02-17 06:48:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:48:25.402783+00:00', 'runtime': '816780.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816784419 ms)
2025-02-17 06:48:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:47:25.400303+00:00', 'runtime': '816720.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816784420 ms)
2025-02-17 06:48:35.658 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816793477 ms)
2025-02-17 06:48:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816793478 ms)
2025-02-17 06:48:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:48:25.400179+00:00'} (T+816793479 ms)
2025-02-17 06:49:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816844405 ms)
2025-02-17 06:49:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816844405 ms)
2025-02-17 06:49:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:49:25.402731+00:00', 'runtime': '816840.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816844408 ms)
2025-02-17 06:49:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:48:25.402783+00:00', 'runtime': '816780.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816844408 ms)
2025-02-17 06:49:35.692 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816853511 ms)
2025-02-17 06:49:35.693 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+816853512 ms)
2025-02-17 06:49:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:49:25.400145+00:00'} (T+816853513 ms)
2025-02-17 06:50:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816904407 ms)
2025-02-17 06:50:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816904407 ms)
2025-02-17 06:50:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:50:25.402699+00:00', 'runtime': '816900.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816904408 ms)
2025-02-17 06:50:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:49:25.402731+00:00', 'runtime': '816840.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816904409 ms)
2025-02-17 06:51:03.499 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+816941318 ms)
2025-02-17 06:51:03.499 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+816941319 ms)
2025-02-17 06:51:03.500 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35160)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+816941319 ms)
2025-02-17 06:51:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+816964406 ms)
2025-02-17 06:51:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+816964407 ms)
2025-02-17 06:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:51:25.400617+00:00', 'runtime': '816960.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816964410 ms)
2025-02-17 06:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:50:25.402699+00:00', 'runtime': '816900.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+816964410 ms)
2025-02-17 06:52:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817024408 ms)
2025-02-17 06:52:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817024408 ms)
2025-02-17 06:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:52:25.400186+00:00', 'runtime': '817020.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817024412 ms)
2025-02-17 06:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:51:25.400617+00:00', 'runtime': '816960.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817024413 ms)
2025-02-17 06:52:35.659 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817033478 ms)
2025-02-17 06:52:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817033479 ms)
2025-02-17 06:52:35.661 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:52:25.400313+00:00'} (T+817033480 ms)
2025-02-17 06:53:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817084410 ms)
2025-02-17 06:53:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817084410 ms)
2025-02-17 06:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:53:25.400198+00:00', 'runtime': '817080.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817084415 ms)
2025-02-17 06:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:52:25.400186+00:00', 'runtime': '817020.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817084415 ms)
2025-02-17 06:53:35.592 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817093412 ms)
2025-02-17 06:53:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817093412 ms)
2025-02-17 06:53:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:53:25.400175+00:00'} (T+817093413 ms)
2025-02-17 06:54:05.801 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817123620 ms)
2025-02-17 06:54:05.801 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817123621 ms)
2025-02-17 06:54:05.802 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35354)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817123621 ms)
2025-02-17 06:54:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817144411 ms)
2025-02-17 06:54:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817144412 ms)
2025-02-17 06:54:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:54:25.400202+00:00', 'runtime': '817140.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.486', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817144413 ms)
2025-02-17 06:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:53:25.400198+00:00', 'runtime': '817080.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20113', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817144414 ms)
2025-02-17 06:54:35.702 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817153522 ms)
2025-02-17 06:54:35.703 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817153522 ms)
2025-02-17 06:54:35.704 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:54:25.400242+00:00'} (T+817153523 ms)
2025-02-17 06:55:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817204413 ms)
2025-02-17 06:55:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817204413 ms)
2025-02-17 06:55:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:55:25.400161+00:00', 'runtime': '817200.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817204414 ms)
2025-02-17 06:55:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:54:25.400202+00:00', 'runtime': '817140.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.486', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817204415 ms)
2025-02-17 06:55:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817213437 ms)
2025-02-17 06:55:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817213438 ms)
2025-02-17 06:55:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:55:25.400088+00:00'} (T+817213439 ms)
2025-02-17 06:55:41.921 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+817219741 ms)
2025-02-17 06:55:41.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+817219741 ms)
2025-02-17 06:55:41.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817219741 ms)
2025-02-17 06:55:47.013 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817224833 ms)
2025-02-17 06:55:47.014 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817224833 ms)
2025-02-17 06:55:47.014 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35470)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817224833 ms)
2025-02-17 06:56:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817264414 ms)
2025-02-17 06:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817264415 ms)
2025-02-17 06:56:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:56:25.400161+00:00', 'runtime': '817260.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.45', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817264416 ms)
2025-02-17 06:56:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:55:25.400161+00:00', 'runtime': '817200.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817264416 ms)
2025-02-17 06:56:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817273435 ms)
2025-02-17 06:56:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817273436 ms)
2025-02-17 06:56:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:56:25.400174+00:00'} (T+817273437 ms)
2025-02-17 06:57:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817324417 ms)
2025-02-17 06:57:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817324418 ms)
2025-02-17 06:57:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:57:25.402850+00:00', 'runtime': '817320.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817324422 ms)
2025-02-17 06:57:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:56:25.400161+00:00', 'runtime': '817260.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.45', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817324422 ms)
2025-02-17 06:57:35.713 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817333532 ms)
2025-02-17 06:57:35.714 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817333533 ms)
2025-02-17 06:57:35.715 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:57:25.400266+00:00'} (T+817333535 ms)
2025-02-17 06:57:57.252 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817355071 ms)
2025-02-17 06:57:57.252 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817355072 ms)
2025-02-17 06:57:57.253 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35614)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817355072 ms)
2025-02-17 06:58:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817384417 ms)
2025-02-17 06:58:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817384417 ms)
2025-02-17 06:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:58:25.402675+00:00', 'runtime': '817380.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817384419 ms)
2025-02-17 06:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:57:25.402850+00:00', 'runtime': '817320.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.3', 'ahl_t': '1587.3', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817384419 ms)
2025-02-17 06:58:35.657 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817393477 ms)
2025-02-17 06:58:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817393477 ms)
2025-02-17 06:58:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:58:25.400101+00:00'} (T+817393478 ms)
2025-02-17 06:59:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817444420 ms)
2025-02-17 06:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817444420 ms)
2025-02-17 06:59:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 06:59:25.402796+00:00', 'runtime': '817440.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817444421 ms)
2025-02-17 06:59:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:58:25.402675+00:00', 'runtime': '817380.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817444422 ms)
2025-02-17 06:59:35.708 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817453527 ms)
2025-02-17 06:59:35.708 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817453527 ms)
2025-02-17 06:59:35.709 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 06:59:25.400160+00:00'} (T+817453528 ms)
2025-02-17 06:59:40.450 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817458269 ms)
2025-02-17 06:59:40.450 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817458269 ms)
2025-02-17 06:59:40.451 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35724)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817458270 ms)
2025-02-17 07:00:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817504421 ms)
2025-02-17 07:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817504422 ms)
2025-02-17 07:00:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:00:25.402758+00:00', 'runtime': '817500.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817504423 ms)
2025-02-17 07:00:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 06:59:25.402796+00:00', 'runtime': '817440.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817504423 ms)
2025-02-17 07:00:35.456 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817513275 ms)
2025-02-17 07:00:35.456 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817513276 ms)
2025-02-17 07:00:35.458 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:00:25.400175+00:00'} (T+817513277 ms)
2025-02-17 07:01:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817564421 ms)
2025-02-17 07:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817564422 ms)
2025-02-17 07:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:01:25.402794+00:00', 'runtime': '817560.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.447', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817564423 ms)
2025-02-17 07:01:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:00:25.402758+00:00', 'runtime': '817500.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817564424 ms)
2025-02-17 07:02:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817624405 ms)
2025-02-17 07:02:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817624405 ms)
2025-02-17 07:02:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:02:25.400147+00:00', 'runtime': '817620.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817624407 ms)
2025-02-17 07:02:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:01:25.402794+00:00', 'runtime': '817560.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.447', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817624407 ms)
2025-02-17 07:03:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817684406 ms)
2025-02-17 07:03:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817684407 ms)
2025-02-17 07:03:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:03:25.400608+00:00', 'runtime': '817680.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817684408 ms)
2025-02-17 07:03:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:02:25.400147+00:00', 'runtime': '817620.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817684408 ms)
2025-02-17 07:04:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817744407 ms)
2025-02-17 07:04:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817744407 ms)
2025-02-17 07:04:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:04:25.402753+00:00', 'runtime': '817740.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817744408 ms)
2025-02-17 07:04:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:03:25.400608+00:00', 'runtime': '817680.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817744409 ms)
2025-02-17 07:04:35.788 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817753607 ms)
2025-02-17 07:04:35.788 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817753608 ms)
2025-02-17 07:04:35.790 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:04:25.400148+00:00'} (T+817753609 ms)
2025-02-17 07:04:42.916 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817760735 ms)
2025-02-17 07:04:42.916 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817760735 ms)
2025-02-17 07:04:42.916 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36044)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817760735 ms)
2025-02-17 07:05:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817804408 ms)
2025-02-17 07:05:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817804409 ms)
2025-02-17 07:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:05:25.402601+00:00', 'runtime': '817800.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817804410 ms)
2025-02-17 07:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:04:25.402753+00:00', 'runtime': '817740.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817804410 ms)
2025-02-17 07:05:29.028 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817806847 ms)
2025-02-17 07:05:29.028 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817806848 ms)
2025-02-17 07:05:29.029 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36094)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817806848 ms)
2025-02-17 07:05:35.726 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817813546 ms)
2025-02-17 07:05:35.727 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817813546 ms)
2025-02-17 07:05:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:05:25.399998+00:00'} (T+817813548 ms)
2025-02-17 07:06:05.151 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817842970 ms)
2025-02-17 07:06:05.151 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817842970 ms)
2025-02-17 07:06:05.151 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36146)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817842970 ms)
2025-02-17 07:06:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817864411 ms)
2025-02-17 07:06:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817864412 ms)
2025-02-17 07:06:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:06:25.402638+00:00', 'runtime': '817860.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817864413 ms)
2025-02-17 07:06:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:05:25.402601+00:00', 'runtime': '817800.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817864413 ms)
2025-02-17 07:06:35.658 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817873477 ms)
2025-02-17 07:06:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817873478 ms)
2025-02-17 07:06:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:06:25.400088+00:00'} (T+817873480 ms)
2025-02-17 07:07:09.281 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817907100 ms)
2025-02-17 07:07:09.281 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817907101 ms)
2025-02-17 07:07:09.282 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36216)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817907101 ms)
2025-02-17 07:07:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817924414 ms)
2025-02-17 07:07:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817924415 ms)
2025-02-17 07:07:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:07:25.402845+00:00', 'runtime': '817920.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817924416 ms)
2025-02-17 07:07:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:06:25.402638+00:00', 'runtime': '817860.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817924416 ms)
2025-02-17 07:07:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817933431 ms)
2025-02-17 07:07:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817933432 ms)
2025-02-17 07:07:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:07:25.400139+00:00'} (T+817933433 ms)
2025-02-17 07:08:12.466 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+817970285 ms)
2025-02-17 07:08:12.466 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+817970286 ms)
2025-02-17 07:08:12.467 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36284)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+817970286 ms)
2025-02-17 07:08:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+817984415 ms)
2025-02-17 07:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+817984415 ms)
2025-02-17 07:08:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:08:25.400195+00:00', 'runtime': '817980.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817984419 ms)
2025-02-17 07:08:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:07:25.402845+00:00', 'runtime': '817920.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+817984419 ms)
2025-02-17 07:08:35.638 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817993458 ms)
2025-02-17 07:08:35.639 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+817993458 ms)
2025-02-17 07:08:35.640 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:08:25.400202+00:00'} (T+817993459 ms)
2025-02-17 07:08:44.580 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818002399 ms)
2025-02-17 07:08:44.580 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818002399 ms)
2025-02-17 07:08:44.580 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36320)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818002400 ms)
2025-02-17 07:09:22.710 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818040529 ms)
2025-02-17 07:09:22.710 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818040530 ms)
2025-02-17 07:09:22.711 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36362)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818040530 ms)
2025-02-17 07:09:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818044414 ms)
2025-02-17 07:09:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818044415 ms)
2025-02-17 07:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:09:25.400194+00:00', 'runtime': '818040.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818044417 ms)
2025-02-17 07:09:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:08:25.400195+00:00', 'runtime': '817980.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818044417 ms)
2025-02-17 07:09:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818053420 ms)
2025-02-17 07:09:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818053421 ms)
2025-02-17 07:09:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:09:25.400187+00:00'} (T+818053425 ms)
2025-02-17 07:10:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818104414 ms)
2025-02-17 07:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818104415 ms)
2025-02-17 07:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:10:25.400573+00:00', 'runtime': '818100.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.471', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818104416 ms)
2025-02-17 07:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:09:25.400194+00:00', 'runtime': '818040.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818104416 ms)
2025-02-17 07:10:42.809 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120628 ms)
2025-02-17 07:10:42.809 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120629 ms)
2025-02-17 07:10:42.810 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120629 ms)
2025-02-17 07:10:42.810 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120629 ms)
2025-02-17 07:10:42.810 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120630 ms)
2025-02-17 07:10:42.811 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120630 ms)
2025-02-17 07:10:42.811 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120630 ms)
2025-02-17 07:10:42.811 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120630 ms)
2025-02-17 07:10:42.811 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120631 ms)
2025-02-17 07:10:42.812 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120631 ms)
2025-02-17 07:10:42.812 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120631 ms)
2025-02-17 07:10:42.812 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120631 ms)
2025-02-17 07:10:42.812 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120632 ms)
2025-02-17 07:10:42.813 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120632 ms)
2025-02-17 07:10:42.813 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120632 ms)
2025-02-17 07:10:42.815 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120634 ms)
2025-02-17 07:10:42.815 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120634 ms)
2025-02-17 07:10:42.816 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120635 ms)
2025-02-17 07:10:42.817 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120636 ms)
2025-02-17 07:10:42.817 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120636 ms)
2025-02-17 07:10:42.818 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120637 ms)
2025-02-17 07:10:42.819 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120638 ms)
2025-02-17 07:10:42.819 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120638 ms)
2025-02-17 07:10:42.820 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120639 ms)
2025-02-17 07:10:42.821 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818120640 ms)
2025-02-17 07:10:42.822 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818120641 ms)
2025-02-17 07:10:42.822 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818120642 ms)
2025-02-17 07:11:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818164413 ms)
2025-02-17 07:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818164414 ms)
2025-02-17 07:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:11:25.400165+00:00', 'runtime': '818160.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.471', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818164415 ms)
2025-02-17 07:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:10:25.400573+00:00', 'runtime': '818100.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.471', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.4', 'ahl_t': '1587.4', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818164415 ms)
2025-02-17 07:12:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818224413 ms)
2025-02-17 07:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818224413 ms)
2025-02-17 07:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:12:25.400169+00:00', 'runtime': '818220.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818224414 ms)
2025-02-17 07:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:11:25.400165+00:00', 'runtime': '818160.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.471', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818224415 ms)
2025-02-17 07:12:36.297 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818234116 ms)
2025-02-17 07:12:36.298 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818234117 ms)
2025-02-17 07:12:36.302 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:12:25.400331+00:00'} (T+818234121 ms)
2025-02-17 07:13:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818284413 ms)
2025-02-17 07:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818284414 ms)
2025-02-17 07:13:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:13:25.400186+00:00', 'runtime': '818280.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818284415 ms)
2025-02-17 07:13:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:12:25.400169+00:00', 'runtime': '818220.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818284415 ms)
2025-02-17 07:13:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818293432 ms)
2025-02-17 07:13:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818293433 ms)
2025-02-17 07:13:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:13:25.400004+00:00'} (T+818293438 ms)
2025-02-17 07:14:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818344415 ms)
2025-02-17 07:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818344416 ms)
2025-02-17 07:14:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:14:25.400214+00:00', 'runtime': '818340.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818344417 ms)
2025-02-17 07:14:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:13:25.400186+00:00', 'runtime': '818280.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818344419 ms)
2025-02-17 07:14:35.687 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818353507 ms)
2025-02-17 07:14:35.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818353507 ms)
2025-02-17 07:14:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:14:25.400233+00:00'} (T+818353514 ms)
2025-02-17 07:14:40.214 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818358033 ms)
2025-02-17 07:14:40.214 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818358033 ms)
2025-02-17 07:14:40.214 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36698)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818358034 ms)
2025-02-17 07:15:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818404416 ms)
2025-02-17 07:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818404416 ms)
2025-02-17 07:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:15:25.400199+00:00', 'runtime': '818400.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818404417 ms)
2025-02-17 07:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:14:25.400214+00:00', 'runtime': '818340.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818404418 ms)
2025-02-17 07:15:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818413432 ms)
2025-02-17 07:15:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818413433 ms)
2025-02-17 07:15:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:15:25.400132+00:00'} (T+818413439 ms)
2025-02-17 07:16:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818464419 ms)
2025-02-17 07:16:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818464420 ms)
2025-02-17 07:16:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:16:25.402754+00:00', 'runtime': '818460.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818464421 ms)
2025-02-17 07:16:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:15:25.400199+00:00', 'runtime': '818400.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818464421 ms)
2025-02-17 07:17:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818524409 ms)
2025-02-17 07:17:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818524409 ms)
2025-02-17 07:17:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:17:25.400610+00:00', 'runtime': '818520.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818524410 ms)
2025-02-17 07:17:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:16:25.402754+00:00', 'runtime': '818460.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818524410 ms)
2025-02-17 07:18:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818584408 ms)
2025-02-17 07:18:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818584409 ms)
2025-02-17 07:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:18:25.400187+00:00', 'runtime': '818580.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818584410 ms)
2025-02-17 07:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:17:25.400610+00:00', 'runtime': '818520.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818584410 ms)
2025-02-17 07:19:23.669 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818641489 ms)
2025-02-17 07:19:23.670 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818641489 ms)
2025-02-17 07:19:23.670 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37008)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818641489 ms)
2025-02-17 07:19:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818644411 ms)
2025-02-17 07:19:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818644412 ms)
2025-02-17 07:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:19:25.400197+00:00', 'runtime': '818640.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818644413 ms)
2025-02-17 07:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:18:25.400187+00:00', 'runtime': '818580.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818644413 ms)
2025-02-17 07:19:33.750 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818651569 ms)
2025-02-17 07:19:33.750 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818651570 ms)
2025-02-17 07:19:33.751 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37020)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818651570 ms)
2025-02-17 07:20:16.902 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818694721 ms)
2025-02-17 07:20:16.902 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818694721 ms)
2025-02-17 07:20:16.902 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37066)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818694721 ms)
2025-02-17 07:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818704413 ms)
2025-02-17 07:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818704413 ms)
2025-02-17 07:20:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:20:25.400168+00:00', 'runtime': '818700.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818704415 ms)
2025-02-17 07:20:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:19:25.400197+00:00', 'runtime': '818640.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818704415 ms)
2025-02-17 07:21:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818764415 ms)
2025-02-17 07:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818764416 ms)
2025-02-17 07:21:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:21:25.400164+00:00', 'runtime': '818760.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818764417 ms)
2025-02-17 07:21:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:20:25.400168+00:00', 'runtime': '818700.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818764417 ms)
2025-02-17 07:21:35.672 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818773491 ms)
2025-02-17 07:21:35.672 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818773492 ms)
2025-02-17 07:21:35.673 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:21:25.413050+00:00'} (T+818773493 ms)
2025-02-17 07:22:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818824416 ms)
2025-02-17 07:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818824417 ms)
2025-02-17 07:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:22:25.402740+00:00', 'runtime': '818820.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818824418 ms)
2025-02-17 07:22:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:21:25.400164+00:00', 'runtime': '818760.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818824418 ms)
2025-02-17 07:22:33.069 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830889 ms)
2025-02-17 07:22:33.070 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830889 ms)
2025-02-17 07:22:33.070 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830889 ms)
2025-02-17 07:22:33.071 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830890 ms)
2025-02-17 07:22:33.071 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830890 ms)
2025-02-17 07:22:33.071 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830890 ms)
2025-02-17 07:22:33.071 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830891 ms)
2025-02-17 07:22:33.072 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830891 ms)
2025-02-17 07:22:33.072 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830891 ms)
2025-02-17 07:22:33.073 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830892 ms)
2025-02-17 07:22:33.073 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830892 ms)
2025-02-17 07:22:33.073 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830893 ms)
2025-02-17 07:22:33.074 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830893 ms)
2025-02-17 07:22:33.074 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830893 ms)
2025-02-17 07:22:33.074 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830894 ms)
2025-02-17 07:22:33.075 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830894 ms)
2025-02-17 07:22:33.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830894 ms)
2025-02-17 07:22:33.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830894 ms)
2025-02-17 07:22:33.076 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830895 ms)
2025-02-17 07:22:33.076 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830895 ms)
2025-02-17 07:22:33.076 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830895 ms)
2025-02-17 07:22:33.076 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830896 ms)
2025-02-17 07:22:33.076 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830896 ms)
2025-02-17 07:22:33.077 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830896 ms)
2025-02-17 07:22:33.080 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830899 ms)
2025-02-17 07:22:33.080 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830899 ms)
2025-02-17 07:22:33.081 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830900 ms)
2025-02-17 07:22:33.083 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+818830902 ms)
2025-02-17 07:22:33.083 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+818830902 ms)
2025-02-17 07:22:33.084 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818830903 ms)
2025-02-17 07:22:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818833434 ms)
2025-02-17 07:22:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+818833434 ms)
2025-02-17 07:22:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:22:25.400129+00:00'} (T+818833435 ms)
2025-02-17 07:23:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818884418 ms)
2025-02-17 07:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818884419 ms)
2025-02-17 07:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:23:25.402772+00:00', 'runtime': '818880.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.507', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818884420 ms)
2025-02-17 07:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:22:25.402740+00:00', 'runtime': '818820.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.216', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818884420 ms)
2025-02-17 07:24:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+818944405 ms)
2025-02-17 07:24:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+818944406 ms)
2025-02-17 07:24:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:24:25.400077+00:00', 'runtime': '818940.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818944407 ms)
2025-02-17 07:24:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:23:25.402772+00:00', 'runtime': '818880.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.507', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.5', 'ahl_t': '1587.5', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+818944407 ms)
2025-02-17 07:25:00.357 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818978177 ms)
2025-02-17 07:25:00.358 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818978177 ms)
2025-02-17 07:25:00.358 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37366)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818978177 ms)
2025-02-17 07:25:13.447 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+818991266 ms)
2025-02-17 07:25:13.447 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+818991266 ms)
2025-02-17 07:25:13.447 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37380)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+818991267 ms)
2025-02-17 07:25:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819004421 ms)
2025-02-17 07:25:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819004421 ms)
2025-02-17 07:25:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:25:25.400187+00:00', 'runtime': '819000.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819004423 ms)
2025-02-17 07:25:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:24:25.400077+00:00', 'runtime': '818940.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819004424 ms)
2025-02-17 07:25:35.722 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819013541 ms)
2025-02-17 07:25:35.722 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819013542 ms)
2025-02-17 07:25:35.725 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:25:25.413048+00:00'} (T+819013545 ms)
2025-02-17 07:26:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819064422 ms)
2025-02-17 07:26:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819064422 ms)
2025-02-17 07:26:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:26:25.402715+00:00', 'runtime': '819060.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819064423 ms)
2025-02-17 07:26:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:25:25.400187+00:00', 'runtime': '819000.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819064424 ms)
2025-02-17 07:26:49.618 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819087437 ms)
2025-02-17 07:26:49.618 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819087437 ms)
2025-02-17 07:26:49.618 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37496)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819087437 ms)
2025-02-17 07:27:21.735 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819119555 ms)
2025-02-17 07:27:21.736 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819119555 ms)
2025-02-17 07:27:21.736 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37530)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819119555 ms)
2025-02-17 07:27:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819124406 ms)
2025-02-17 07:27:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819124406 ms)
2025-02-17 07:27:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:27:25.400036+00:00', 'runtime': '819120.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819124407 ms)
2025-02-17 07:27:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:26:25.402715+00:00', 'runtime': '819060.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.456', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819124408 ms)
2025-02-17 07:28:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819184423 ms)
2025-02-17 07:28:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819184424 ms)
2025-02-17 07:28:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:28:25.400191+00:00', 'runtime': '819180.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819184425 ms)
2025-02-17 07:28:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:27:25.400036+00:00', 'runtime': '819120.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819184425 ms)
2025-02-17 07:29:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819244408 ms)
2025-02-17 07:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819244408 ms)
2025-02-17 07:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:29:25.400251+00:00', 'runtime': '819240.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819244409 ms)
2025-02-17 07:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:28:25.400191+00:00', 'runtime': '819180.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819244410 ms)
2025-02-17 07:29:27.959 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819245778 ms)
2025-02-17 07:29:27.959 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819245778 ms)
2025-02-17 07:29:27.960 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37664)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819245779 ms)
2025-02-17 07:29:27.998 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819245818 ms)
2025-02-17 07:29:27.999 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819245818 ms)
2025-02-17 07:29:27.999 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37666)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819245818 ms)
2025-02-17 07:30:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819304412 ms)
2025-02-17 07:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819304412 ms)
2025-02-17 07:30:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:30:25.400201+00:00', 'runtime': '819300.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819304416 ms)
2025-02-17 07:30:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:29:25.400251+00:00', 'runtime': '819240.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819304417 ms)
2025-02-17 07:30:52.169 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819329988 ms)
2025-02-17 07:30:52.169 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819329989 ms)
2025-02-17 07:30:52.170 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37756)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819329989 ms)
2025-02-17 07:31:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819364414 ms)
2025-02-17 07:31:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819364414 ms)
2025-02-17 07:31:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:31:25.400171+00:00', 'runtime': '819360.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819364419 ms)
2025-02-17 07:31:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:30:25.400201+00:00', 'runtime': '819300.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819364419 ms)
2025-02-17 07:31:35.444 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819373263 ms)
2025-02-17 07:31:35.445 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819373264 ms)
2025-02-17 07:31:35.446 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:31:25.405062+00:00'} (T+819373265 ms)
2025-02-17 07:31:54.311 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819392130 ms)
2025-02-17 07:31:54.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819392130 ms)
2025-02-17 07:31:54.312 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37824)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819392131 ms)
2025-02-17 07:32:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819424415 ms)
2025-02-17 07:32:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819424416 ms)
2025-02-17 07:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:32:25.401172+00:00', 'runtime': '819420.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819424417 ms)
2025-02-17 07:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:31:25.400171+00:00', 'runtime': '819360.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819424417 ms)
2025-02-17 07:33:08.481 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819466300 ms)
2025-02-17 07:33:08.481 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819466300 ms)
2025-02-17 07:33:08.482 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37904)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819466301 ms)
2025-02-17 07:33:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819484401 ms)
2025-02-17 07:33:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819484402 ms)
2025-02-17 07:33:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:33:25.400037+00:00', 'runtime': '819480.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819484403 ms)
2025-02-17 07:33:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:32:25.401172+00:00', 'runtime': '819420.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819484403 ms)
2025-02-17 07:34:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819544420 ms)
2025-02-17 07:34:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819544420 ms)
2025-02-17 07:34:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:34:25.400212+00:00', 'runtime': '819540.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819544421 ms)
2025-02-17 07:34:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:33:25.400037+00:00', 'runtime': '819480.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819544422 ms)
2025-02-17 07:34:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819553434 ms)
2025-02-17 07:34:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819553435 ms)
2025-02-17 07:34:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:34:25.405065+00:00'} (T+819553438 ms)
2025-02-17 07:35:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819604420 ms)
2025-02-17 07:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819604420 ms)
2025-02-17 07:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:35:25.400167+00:00', 'runtime': '819600.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819604422 ms)
2025-02-17 07:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:34:25.400212+00:00', 'runtime': '819540.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.465', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819604423 ms)
2025-02-17 07:35:35.799 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819613618 ms)
2025-02-17 07:35:35.799 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819613619 ms)
2025-02-17 07:35:35.800 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:35:25.413123+00:00'} (T+819613620 ms)
2025-02-17 07:36:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819664422 ms)
2025-02-17 07:36:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819664422 ms)
2025-02-17 07:36:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:36:25.402598+00:00', 'runtime': '819660.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819664423 ms)
2025-02-17 07:36:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:35:25.400167+00:00', 'runtime': '819600.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.468', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.213', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819664425 ms)
2025-02-17 07:36:35.652 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819673472 ms)
2025-02-17 07:36:35.653 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819673472 ms)
2025-02-17 07:36:35.654 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:36:25.400003+00:00'} (T+819673473 ms)
2025-02-17 07:37:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819724407 ms)
2025-02-17 07:37:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819724407 ms)
2025-02-17 07:37:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:37:25.402740+00:00', 'runtime': '819720.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819724408 ms)
2025-02-17 07:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:36:25.402598+00:00', 'runtime': '819660.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.477', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.6', 'ahl_t': '1587.6', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819724409 ms)
2025-02-17 07:37:35.656 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819733475 ms)
2025-02-17 07:37:35.657 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819733476 ms)
2025-02-17 07:37:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:37:25.400125+00:00'} (T+819733478 ms)
2025-02-17 07:38:26.606 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819784425 ms)
2025-02-17 07:38:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819784425 ms)
2025-02-17 07:38:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:38:25.401127+00:00', 'runtime': '819780.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819784427 ms)
2025-02-17 07:38:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:37:25.402740+00:00', 'runtime': '819720.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819784428 ms)
2025-02-17 07:38:35.867 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819793686 ms)
2025-02-17 07:38:35.868 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819793687 ms)
2025-02-17 07:38:35.869 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:38:25.400135+00:00'} (T+819793688 ms)
2025-02-17 07:38:59.973 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+819817792 ms)
2025-02-17 07:38:59.973 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+819817792 ms)
2025-02-17 07:38:59.973 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38288)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+819817793 ms)
2025-02-17 07:39:28.463 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819846282 ms)
2025-02-17 07:39:28.463 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819846282 ms)
2025-02-17 07:39:28.464 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:39:25.400175+00:00', 'runtime': '819840.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819846284 ms)
2025-02-17 07:39:28.465 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:38:25.401127+00:00', 'runtime': '819780.002', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819846284 ms)
2025-02-17 07:39:35.672 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819853492 ms)
2025-02-17 07:39:35.673 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819853492 ms)
2025-02-17 07:39:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:39:25.413120+00:00'} (T+819853493 ms)
2025-02-17 07:40:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819904414 ms)
2025-02-17 07:40:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819904415 ms)
2025-02-17 07:40:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:40:25.402766+00:00', 'runtime': '819900.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819904417 ms)
2025-02-17 07:40:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:39:25.400175+00:00', 'runtime': '819840.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.483', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819904417 ms)
2025-02-17 07:40:35.628 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819913448 ms)
2025-02-17 07:40:35.629 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819913448 ms)
2025-02-17 07:40:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:40:25.400131+00:00'} (T+819913449 ms)
2025-02-17 07:41:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+819964416 ms)
2025-02-17 07:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+819964416 ms)
2025-02-17 07:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:41:25.402798+00:00', 'runtime': '819960.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819964417 ms)
2025-02-17 07:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:40:25.402766+00:00', 'runtime': '819900.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+819964418 ms)
2025-02-17 07:41:35.624 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819973443 ms)
2025-02-17 07:41:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+819973443 ms)
2025-02-17 07:41:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:41:25.400159+00:00'} (T+819973445 ms)
2025-02-17 07:42:02.274 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+820000093 ms)
2025-02-17 07:42:02.274 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+820000094 ms)
2025-02-17 07:42:02.275 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38484)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820000094 ms)
2025-02-17 07:42:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820024416 ms)
2025-02-17 07:42:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820024416 ms)
2025-02-17 07:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:42:25.402787+00:00', 'runtime': '820020.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820024417 ms)
2025-02-17 07:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:41:25.402798+00:00', 'runtime': '819960.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.495', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820024418 ms)
2025-02-17 07:42:31.400 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+820029220 ms)
2025-02-17 07:42:31.401 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+820029220 ms)
2025-02-17 07:42:31.401 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38514)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820029220 ms)
2025-02-17 07:42:36.092 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820033911 ms)
2025-02-17 07:42:36.092 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820033911 ms)
2025-02-17 07:42:36.093 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:42:25.400176+00:00'} (T+820033912 ms)
2025-02-17 07:43:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820084420 ms)
2025-02-17 07:43:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820084420 ms)
2025-02-17 07:43:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:43:25.402763+00:00', 'runtime': '820080.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820084422 ms)
2025-02-17 07:43:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:42:25.402787+00:00', 'runtime': '820020.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820084423 ms)
2025-02-17 07:43:35.846 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820093665 ms)
2025-02-17 07:43:35.847 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820093667 ms)
2025-02-17 07:43:35.853 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:43:25.400186+00:00'} (T+820093672 ms)
2025-02-17 07:44:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820144421 ms)
2025-02-17 07:44:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820144422 ms)
2025-02-17 07:44:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:44:25.402849+00:00', 'runtime': '820140.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820144423 ms)
2025-02-17 07:44:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:43:25.402763+00:00', 'runtime': '820080.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820144423 ms)
2025-02-17 07:44:35.761 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820153580 ms)
2025-02-17 07:44:35.761 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820153581 ms)
2025-02-17 07:44:35.762 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:44:25.400210+00:00'} (T+820153582 ms)
2025-02-17 07:45:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820204408 ms)
2025-02-17 07:45:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820204408 ms)
2025-02-17 07:45:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:45:25.402699+00:00', 'runtime': '820200.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820204409 ms)
2025-02-17 07:45:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:44:25.402849+00:00', 'runtime': '820140.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.492', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820204410 ms)
2025-02-17 07:45:36.135 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820213954 ms)
2025-02-17 07:45:36.136 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820213955 ms)
2025-02-17 07:45:36.137 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:45:25.400028+00:00'} (T+820213956 ms)
2025-02-17 07:46:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820264409 ms)
2025-02-17 07:46:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820264409 ms)
2025-02-17 07:46:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:46:25.402713+00:00', 'runtime': '820260.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820264410 ms)
2025-02-17 07:46:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:45:25.402699+00:00', 'runtime': '820200.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.204', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820264411 ms)
2025-02-17 07:46:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820273434 ms)
2025-02-17 07:46:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820273434 ms)
2025-02-17 07:46:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:46:25.400115+00:00'} (T+820273435 ms)
2025-02-17 07:46:55.724 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+820293543 ms)
2025-02-17 07:46:55.724 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+820293543 ms)
2025-02-17 07:46:55.725 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820293544 ms)
2025-02-17 07:46:55.726 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+820293545 ms)
2025-02-17 07:46:55.726 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+820293545 ms)
2025-02-17 07:46:55.726 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820293546 ms)
2025-02-17 07:47:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820324408 ms)
2025-02-17 07:47:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820324409 ms)
2025-02-17 07:47:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:47:25.402776+00:00', 'runtime': '820320.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820324410 ms)
2025-02-17 07:47:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:46:25.402713+00:00', 'runtime': '820260.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820324410 ms)
2025-02-17 07:47:35.913 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820333732 ms)
2025-02-17 07:47:35.913 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820333733 ms)
2025-02-17 07:47:35.914 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:47:25.400172+00:00'} (T+820333734 ms)
2025-02-17 07:48:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820384410 ms)
2025-02-17 07:48:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820384410 ms)
2025-02-17 07:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:48:25.400161+00:00', 'runtime': '820380.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820384411 ms)
2025-02-17 07:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:47:25.402776+00:00', 'runtime': '820320.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820384411 ms)
2025-02-17 07:48:35.658 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820393478 ms)
2025-02-17 07:48:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820393478 ms)
2025-02-17 07:48:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:48:25.400400+00:00'} (T+820393480 ms)
2025-02-17 07:49:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820444407 ms)
2025-02-17 07:49:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820444408 ms)
2025-02-17 07:49:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:49:25.400178+00:00', 'runtime': '820440.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820444410 ms)
2025-02-17 07:49:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:48:25.400161+00:00', 'runtime': '820380.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820444410 ms)
2025-02-17 07:50:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820504407 ms)
2025-02-17 07:50:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820504407 ms)
2025-02-17 07:50:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:50:25.400194+00:00', 'runtime': '820500.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820504409 ms)
2025-02-17 07:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:49:25.400178+00:00', 'runtime': '820440.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.7', 'ahl_t': '1587.7', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820504410 ms)
2025-02-17 07:50:41.090 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+820518910 ms)
2025-02-17 07:50:41.091 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+820518910 ms)
2025-02-17 07:50:41.091 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39046)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820518910 ms)
2025-02-17 07:51:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820564407 ms)
2025-02-17 07:51:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820564408 ms)
2025-02-17 07:51:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:51:25.400201+00:00', 'runtime': '820560.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820564409 ms)
2025-02-17 07:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:50:25.400194+00:00', 'runtime': '820500.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820564409 ms)
2025-02-17 07:51:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820573428 ms)
2025-02-17 07:51:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820573429 ms)
2025-02-17 07:51:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:51:25.400333+00:00'} (T+820573430 ms)
2025-02-17 07:52:10.267 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+820608086 ms)
2025-02-17 07:52:10.267 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+820608086 ms)
2025-02-17 07:52:10.268 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39142)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820608087 ms)
2025-02-17 07:52:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820624407 ms)
2025-02-17 07:52:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820624407 ms)
2025-02-17 07:52:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:52:25.400172+00:00', 'runtime': '820620.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820624408 ms)
2025-02-17 07:52:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:51:25.400201+00:00', 'runtime': '820560.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820624409 ms)
2025-02-17 07:52:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820633428 ms)
2025-02-17 07:52:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820633429 ms)
2025-02-17 07:52:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:52:25.400179+00:00'} (T+820633436 ms)
2025-02-17 07:53:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820684406 ms)
2025-02-17 07:53:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820684407 ms)
2025-02-17 07:53:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:53:25.400262+00:00', 'runtime': '820680.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820684408 ms)
2025-02-17 07:53:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:52:25.400172+00:00', 'runtime': '820620.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820684411 ms)
2025-02-17 07:53:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820693425 ms)
2025-02-17 07:53:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820693426 ms)
2025-02-17 07:53:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:53:25.400269+00:00'} (T+820693427 ms)
2025-02-17 07:54:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820744406 ms)
2025-02-17 07:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820744407 ms)
2025-02-17 07:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:54:25.400269+00:00', 'runtime': '820740.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820744408 ms)
2025-02-17 07:54:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:53:25.400262+00:00', 'runtime': '820680.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20114', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820744408 ms)
2025-02-17 07:54:34.484 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+820752303 ms)
2025-02-17 07:54:34.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+820752304 ms)
2025-02-17 07:54:34.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39298)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820752304 ms)
2025-02-17 07:54:35.758 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820753577 ms)
2025-02-17 07:54:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820753578 ms)
2025-02-17 07:54:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:54:25.400269+00:00'} (T+820753579 ms)
2025-02-17 07:54:44.562 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+820762382 ms)
2025-02-17 07:54:44.563 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+820762382 ms)
2025-02-17 07:54:44.563 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39310)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820762382 ms)
2025-02-17 07:55:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820804409 ms)
2025-02-17 07:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820804409 ms)
2025-02-17 07:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:55:25.400174+00:00', 'runtime': '820800.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820804411 ms)
2025-02-17 07:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:54:25.400269+00:00', 'runtime': '820740.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820804412 ms)
2025-02-17 07:55:35.763 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820813583 ms)
2025-02-17 07:55:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820813583 ms)
2025-02-17 07:55:35.765 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:55:25.400025+00:00'} (T+820813584 ms)
2025-02-17 07:56:12.670 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+820850489 ms)
2025-02-17 07:56:12.670 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+820850489 ms)
2025-02-17 07:56:12.671 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820850490 ms)
2025-02-17 07:56:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820864411 ms)
2025-02-17 07:56:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820864411 ms)
2025-02-17 07:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:56:25.400180+00:00', 'runtime': '820860.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820864414 ms)
2025-02-17 07:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:55:25.400174+00:00', 'runtime': '820800.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820864415 ms)
2025-02-17 07:57:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820924411 ms)
2025-02-17 07:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820924411 ms)
2025-02-17 07:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:57:25.400983+00:00', 'runtime': '820920.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820924415 ms)
2025-02-17 07:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:56:25.400180+00:00', 'runtime': '820860.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820924416 ms)
2025-02-17 07:57:44.869 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+820942688 ms)
2025-02-17 07:57:44.869 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+820942689 ms)
2025-02-17 07:57:44.870 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39512)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+820942689 ms)
2025-02-17 07:58:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+820984411 ms)
2025-02-17 07:58:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+820984411 ms)
2025-02-17 07:58:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:58:25.400135+00:00', 'runtime': '820980.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820984412 ms)
2025-02-17 07:58:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:57:25.400983+00:00', 'runtime': '820920.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.501', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+820984415 ms)
2025-02-17 07:58:35.596 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820993415 ms)
2025-02-17 07:58:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+820993416 ms)
2025-02-17 07:58:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:58:25.400324+00:00'} (T+820993417 ms)
2025-02-17 07:59:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821044412 ms)
2025-02-17 07:59:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821044413 ms)
2025-02-17 07:59:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 07:59:25.400176+00:00', 'runtime': '821040.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821044414 ms)
2025-02-17 07:59:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:58:25.400135+00:00', 'runtime': '820980.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.498', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821044414 ms)
2025-02-17 07:59:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821053429 ms)
2025-02-17 07:59:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821053430 ms)
2025-02-17 07:59:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 07:59:25.400203+00:00'} (T+821053431 ms)
2025-02-17 08:00:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821104425 ms)
2025-02-17 08:00:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821104425 ms)
2025-02-17 08:00:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:00:25.400197+00:00', 'runtime': '821100.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821104426 ms)
2025-02-17 08:00:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 07:59:25.400176+00:00', 'runtime': '821040.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.489', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821104427 ms)
2025-02-17 08:00:35.645 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821113465 ms)
2025-02-17 08:00:35.646 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821113465 ms)
2025-02-17 08:00:35.647 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:00:25.400350+00:00'} (T+821113466 ms)
2025-02-17 08:01:04.201 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821142020 ms)
2025-02-17 08:01:04.201 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821142021 ms)
2025-02-17 08:01:04.202 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39726)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821142021 ms)
2025-02-17 08:01:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821164407 ms)
2025-02-17 08:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821164408 ms)
2025-02-17 08:01:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:01:25.402747+00:00', 'runtime': '821160.003', 'adc_vb_f': '12.213', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821164410 ms)
2025-02-17 08:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:00:25.400197+00:00', 'runtime': '821100.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821164410 ms)
2025-02-17 08:01:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821173438 ms)
2025-02-17 08:01:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821173439 ms)
2025-02-17 08:01:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:01:25.400179+00:00'} (T+821173440 ms)
2025-02-17 08:02:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821224408 ms)
2025-02-17 08:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821224409 ms)
2025-02-17 08:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:02:25.402866+00:00', 'runtime': '821220.003', 'adc_vb_f': '12.213', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821224413 ms)
2025-02-17 08:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:01:25.402747+00:00', 'runtime': '821160.003', 'adc_vb_f': '12.213', 'adc_va_f': '1.48', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.8', 'ahl_t': '1587.8', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821224413 ms)
2025-02-17 08:02:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821233435 ms)
2025-02-17 08:02:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821233435 ms)
2025-02-17 08:02:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:02:25.400198+00:00'} (T+821233436 ms)
2025-02-17 08:03:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821284409 ms)
2025-02-17 08:03:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821284409 ms)
2025-02-17 08:03:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:03:25.402764+00:00', 'runtime': '821280.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821284410 ms)
2025-02-17 08:03:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:02:25.402866+00:00', 'runtime': '821220.003', 'adc_vb_f': '12.213', 'adc_va_f': '1.474', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821284410 ms)
2025-02-17 08:03:34.433 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821292252 ms)
2025-02-17 08:03:34.433 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821292252 ms)
2025-02-17 08:03:34.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39888)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821292253 ms)
2025-02-17 08:03:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821293444 ms)
2025-02-17 08:03:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821293445 ms)
2025-02-17 08:03:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:03:25.400173+00:00'} (T+821293446 ms)
2025-02-17 08:03:40.492 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821298312 ms)
2025-02-17 08:03:40.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821298312 ms)
2025-02-17 08:03:40.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39896)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821298312 ms)
2025-02-17 08:04:26.608 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821344427 ms)
2025-02-17 08:04:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821344428 ms)
2025-02-17 08:04:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:04:25.402771+00:00', 'runtime': '821340.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821344429 ms)
2025-02-17 08:04:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:03:25.402764+00:00', 'runtime': '821280.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.462', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821344429 ms)
2025-02-17 08:04:35.692 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821353511 ms)
2025-02-17 08:04:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821353512 ms)
2025-02-17 08:04:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:04:25.400160+00:00'} (T+821353513 ms)
2025-02-17 08:05:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821404410 ms)
2025-02-17 08:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821404411 ms)
2025-02-17 08:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:05:25.400155+00:00', 'runtime': '821400.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.453', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821404412 ms)
2025-02-17 08:05:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:04:25.402771+00:00', 'runtime': '821340.003', 'adc_vb_f': '12.216', 'adc_va_f': '1.459', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821404412 ms)
2025-02-17 08:05:31.682 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821409501 ms)
2025-02-17 08:05:31.682 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821409501 ms)
2025-02-17 08:05:31.682 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40016)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821409502 ms)
2025-02-17 08:05:32.780 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821410600 ms)
2025-02-17 08:05:32.781 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821410600 ms)
2025-02-17 08:05:32.781 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40018)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821410600 ms)
2025-02-17 08:06:02.817 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+821440637 ms)
2025-02-17 08:06:02.818 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+821440637 ms)
2025-02-17 08:06:02.818 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821440637 ms)
2025-02-17 08:06:02.819 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+821440638 ms)
2025-02-17 08:06:02.819 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+821440638 ms)
2025-02-17 08:06:02.819 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821440639 ms)
2025-02-17 08:06:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821464417 ms)
2025-02-17 08:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821464417 ms)
2025-02-17 08:06:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:06:25.400802+00:00', 'runtime': '821460.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.447', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821464421 ms)
2025-02-17 08:06:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:05:25.400155+00:00', 'runtime': '821400.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.453', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.216', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821464421 ms)
2025-02-17 08:07:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821524414 ms)
2025-02-17 08:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821524415 ms)
2025-02-17 08:07:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:07:25.402751+00:00', 'runtime': '821520.003', 'adc_vb_f': '12.213', 'adc_va_f': '1.425', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821524416 ms)
2025-02-17 08:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:06:25.400802+00:00', 'runtime': '821460.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.447', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821524419 ms)
2025-02-17 08:07:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821533432 ms)
2025-02-17 08:07:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821533433 ms)
2025-02-17 08:07:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:07:25.400120+00:00'} (T+821533434 ms)
2025-02-17 08:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821584414 ms)
2025-02-17 08:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821584415 ms)
2025-02-17 08:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:08:25.400220+00:00', 'runtime': '821580.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821584416 ms)
2025-02-17 08:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:07:25.402751+00:00', 'runtime': '821520.003', 'adc_vb_f': '12.213', 'adc_va_f': '1.425', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821584416 ms)
2025-02-17 08:08:32.089 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821589909 ms)
2025-02-17 08:08:32.090 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821589909 ms)
2025-02-17 08:08:32.090 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40216)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821589910 ms)
2025-02-17 08:08:54.208 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821612027 ms)
2025-02-17 08:08:54.208 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821612028 ms)
2025-02-17 08:08:54.209 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40244)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821612028 ms)
2025-02-17 08:09:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821644415 ms)
2025-02-17 08:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821644416 ms)
2025-02-17 08:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:09:25.400155+00:00', 'runtime': '821640.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821644417 ms)
2025-02-17 08:09:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:08:25.400220+00:00', 'runtime': '821580.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821644417 ms)
2025-02-17 08:09:48.277 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+821666097 ms)
2025-02-17 08:09:48.278 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+821666097 ms)
2025-02-17 08:09:48.278 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821666097 ms)
2025-02-17 08:09:48.279 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+821666098 ms)
2025-02-17 08:09:48.279 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+821666098 ms)
2025-02-17 08:09:48.280 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821666099 ms)
2025-02-17 08:10:22.390 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821700209 ms)
2025-02-17 08:10:22.390 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821700209 ms)
2025-02-17 08:10:22.390 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40336)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821700210 ms)
2025-02-17 08:10:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821704416 ms)
2025-02-17 08:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821704417 ms)
2025-02-17 08:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:10:25.400200+00:00', 'runtime': '821700.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821704418 ms)
2025-02-17 08:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:09:25.400155+00:00', 'runtime': '821640.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821704418 ms)
2025-02-17 08:10:47.505 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821725324 ms)
2025-02-17 08:10:47.505 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821725324 ms)
2025-02-17 08:10:47.506 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40364)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821725325 ms)
2025-02-17 08:11:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821764401 ms)
2025-02-17 08:11:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821764402 ms)
2025-02-17 08:11:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:11:25.400165+00:00', 'runtime': '821760.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.392', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821764403 ms)
2025-02-17 08:11:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:10:25.400200+00:00', 'runtime': '821700.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821764403 ms)
2025-02-17 08:12:26.581 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821824400 ms)
2025-02-17 08:12:26.581 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821824400 ms)
2025-02-17 08:12:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:12:25.400184+00:00', 'runtime': '821820.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.392', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821824401 ms)
2025-02-17 08:12:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:11:25.400165+00:00', 'runtime': '821760.001', 'adc_vb_f': '12.216', 'adc_va_f': '1.392', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821824402 ms)
2025-02-17 08:13:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821884403 ms)
2025-02-17 08:13:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821884403 ms)
2025-02-17 08:13:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:13:25.400184+00:00', 'runtime': '821880.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.398', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821884404 ms)
2025-02-17 08:13:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:12:25.400184+00:00', 'runtime': '821820.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.392', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821884405 ms)
2025-02-17 08:13:35.597 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821893416 ms)
2025-02-17 08:13:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821893417 ms)
2025-02-17 08:13:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:13:25.400343+00:00'} (T+821893418 ms)
2025-02-17 08:14:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+821944406 ms)
2025-02-17 08:14:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+821944406 ms)
2025-02-17 08:14:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:14:25.400280+00:00', 'runtime': '821940.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.395', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821944407 ms)
2025-02-17 08:14:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:13:25.400184+00:00', 'runtime': '821880.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.398', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.201', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+821944408 ms)
2025-02-17 08:14:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821953418 ms)
2025-02-17 08:14:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+821953419 ms)
2025-02-17 08:14:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:14:25.400286+00:00'} (T+821953423 ms)
2025-02-17 08:14:46.906 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+821964725 ms)
2025-02-17 08:14:46.906 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+821964725 ms)
2025-02-17 08:14:46.907 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40616)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+821964726 ms)
2025-02-17 08:15:22.996 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822000815 ms)
2025-02-17 08:15:22.996 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822000816 ms)
2025-02-17 08:15:22.997 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40656)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822000816 ms)
2025-02-17 08:15:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822004407 ms)
2025-02-17 08:15:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822004408 ms)
2025-02-17 08:15:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:15:25.400221+00:00', 'runtime': '822000.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.395', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822004409 ms)
2025-02-17 08:15:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:14:25.400280+00:00', 'runtime': '821940.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.395', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1587.9', 'ahl_t': '1587.9', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822004409 ms)
2025-02-17 08:15:35.597 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822013417 ms)
2025-02-17 08:15:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822013417 ms)
2025-02-17 08:15:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:15:25.400231+00:00'} (T+822013419 ms)
2025-02-17 08:15:55.105 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822032924 ms)
2025-02-17 08:15:55.105 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822032925 ms)
2025-02-17 08:15:55.106 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40704)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822032925 ms)
2025-02-17 08:16:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822064410 ms)
2025-02-17 08:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822064410 ms)
2025-02-17 08:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:16:25.400178+00:00', 'runtime': '822060.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.404', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822064411 ms)
2025-02-17 08:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:15:25.400221+00:00', 'runtime': '822000.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.395', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822064412 ms)
2025-02-17 08:16:35.641 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822073460 ms)
2025-02-17 08:16:35.642 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822073461 ms)
2025-02-17 08:16:35.648 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:16:25.400201+00:00'} (T+822073467 ms)
2025-02-17 08:17:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822124411 ms)
2025-02-17 08:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822124411 ms)
2025-02-17 08:17:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:17:25.400191+00:00', 'runtime': '822120.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822124412 ms)
2025-02-17 08:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:16:25.400178+00:00', 'runtime': '822060.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.404', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822124413 ms)
2025-02-17 08:17:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822133426 ms)
2025-02-17 08:17:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822133427 ms)
2025-02-17 08:17:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:17:25.400174+00:00'} (T+822133432 ms)
2025-02-17 08:17:57.336 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822155155 ms)
2025-02-17 08:17:57.336 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822155155 ms)
2025-02-17 08:17:57.337 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40836)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822155156 ms)
2025-02-17 08:18:14.435 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822172255 ms)
2025-02-17 08:18:14.436 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822172255 ms)
2025-02-17 08:18:14.436 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40854)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822172255 ms)
2025-02-17 08:18:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822184415 ms)
2025-02-17 08:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822184415 ms)
2025-02-17 08:18:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:18:25.400169+00:00', 'runtime': '822180.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822184417 ms)
2025-02-17 08:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:17:25.400191+00:00', 'runtime': '822120.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822184417 ms)
2025-02-17 08:18:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822193420 ms)
2025-02-17 08:18:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822193421 ms)
2025-02-17 08:18:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:18:25.400161+00:00'} (T+822193426 ms)
2025-02-17 08:19:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822244413 ms)
2025-02-17 08:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822244413 ms)
2025-02-17 08:19:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:19:25.400271+00:00', 'runtime': '822240.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822244414 ms)
2025-02-17 08:19:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:18:25.400169+00:00', 'runtime': '822180.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822244415 ms)
2025-02-17 08:19:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822253424 ms)
2025-02-17 08:19:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822253425 ms)
2025-02-17 08:19:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:19:25.400271+00:00'} (T+822253431 ms)
2025-02-17 08:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822304412 ms)
2025-02-17 08:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822304413 ms)
2025-02-17 08:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:20:25.400176+00:00', 'runtime': '822300.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822304414 ms)
2025-02-17 08:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:19:25.400271+00:00', 'runtime': '822240.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822304414 ms)
2025-02-17 08:20:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822313422 ms)
2025-02-17 08:20:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822313423 ms)
2025-02-17 08:20:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:20:25.400202+00:00'} (T+822313430 ms)
2025-02-17 08:21:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822364413 ms)
2025-02-17 08:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822364414 ms)
2025-02-17 08:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:21:25.400183+00:00', 'runtime': '822360.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822364415 ms)
2025-02-17 08:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:20:25.400176+00:00', 'runtime': '822300.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822364415 ms)
2025-02-17 08:21:33.733 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822371553 ms)
2025-02-17 08:21:33.734 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822371553 ms)
2025-02-17 08:21:33.734 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41066)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822371553 ms)
2025-02-17 08:21:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822373425 ms)
2025-02-17 08:21:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822373426 ms)
2025-02-17 08:21:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:21:25.400208+00:00'} (T+822373435 ms)
2025-02-17 08:21:57.835 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822395654 ms)
2025-02-17 08:21:57.835 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822395654 ms)
2025-02-17 08:21:57.835 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41096)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822395654 ms)
2025-02-17 08:22:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822424416 ms)
2025-02-17 08:22:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822424416 ms)
2025-02-17 08:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:22:25.400184+00:00', 'runtime': '822420.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822424417 ms)
2025-02-17 08:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:21:25.400183+00:00', 'runtime': '822360.001', 'adc_vb_f': '12.213', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822424418 ms)
2025-02-17 08:22:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822433426 ms)
2025-02-17 08:22:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822433427 ms)
2025-02-17 08:22:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:22:25.400206+00:00'} (T+822433434 ms)
2025-02-17 08:23:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822484416 ms)
2025-02-17 08:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822484417 ms)
2025-02-17 08:23:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:23:25.402787+00:00', 'runtime': '822480.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822484421 ms)
2025-02-17 08:23:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:22:25.400184+00:00', 'runtime': '822420.001', 'adc_vb_f': '12.21', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822484422 ms)
2025-02-17 08:23:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822493438 ms)
2025-02-17 08:23:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822493439 ms)
2025-02-17 08:23:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:23:25.400198+00:00'} (T+822493444 ms)
2025-02-17 08:23:42.023 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822499843 ms)
2025-02-17 08:23:42.024 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822499843 ms)
2025-02-17 08:23:42.024 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41208)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822499843 ms)
2025-02-17 08:24:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822544419 ms)
2025-02-17 08:24:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822544419 ms)
2025-02-17 08:24:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:24:25.402755+00:00', 'runtime': '822540.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.407', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822544423 ms)
2025-02-17 08:24:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:23:25.402787+00:00', 'runtime': '822480.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.41', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.21', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822544424 ms)
2025-02-17 08:24:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822553426 ms)
2025-02-17 08:24:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822553427 ms)
2025-02-17 08:24:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:24:25.400162+00:00'} (T+822553431 ms)
2025-02-17 08:25:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822604406 ms)
2025-02-17 08:25:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822604406 ms)
2025-02-17 08:25:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:25:25.402858+00:00', 'runtime': '822600.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.41', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822604411 ms)
2025-02-17 08:25:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:24:25.402755+00:00', 'runtime': '822540.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.407', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822604411 ms)
2025-02-17 08:25:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822613428 ms)
2025-02-17 08:25:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822613429 ms)
2025-02-17 08:25:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:25:25.400212+00:00'} (T+822613433 ms)
2025-02-17 08:25:51.236 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822629055 ms)
2025-02-17 08:25:51.236 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822629055 ms)
2025-02-17 08:25:51.236 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41358)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822629056 ms)
2025-02-17 08:25:56.307 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822634127 ms)
2025-02-17 08:25:56.308 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822634127 ms)
2025-02-17 08:25:56.308 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41364)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822634127 ms)
2025-02-17 08:26:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822664424 ms)
2025-02-17 08:26:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822664424 ms)
2025-02-17 08:26:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:26:25.402792+00:00', 'runtime': '822660.003', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822664426 ms)
2025-02-17 08:26:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:25:25.402858+00:00', 'runtime': '822600.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.41', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822664427 ms)
2025-02-17 08:26:35.710 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822673530 ms)
2025-02-17 08:26:35.711 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822673530 ms)
2025-02-17 08:26:35.715 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:26:25.400186+00:00'} (T+822673534 ms)
2025-02-17 08:27:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822724411 ms)
2025-02-17 08:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822724411 ms)
2025-02-17 08:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:27:25.402792+00:00', 'runtime': '822720.003', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822724415 ms)
2025-02-17 08:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:26:25.402792+00:00', 'runtime': '822660.003', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822724415 ms)
2025-02-17 08:27:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822733439 ms)
2025-02-17 08:27:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822733439 ms)
2025-02-17 08:27:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:27:25.400171+00:00'} (T+822733442 ms)
2025-02-17 08:28:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822784415 ms)
2025-02-17 08:28:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822784415 ms)
2025-02-17 08:28:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:28:25.402800+00:00', 'runtime': '822780.003', 'adc_vb_f': '12.21', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822784420 ms)
2025-02-17 08:28:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:27:25.402792+00:00', 'runtime': '822720.003', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.0', 'ahl_t': '1588.0', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822784420 ms)
2025-02-17 08:28:35.680 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822793499 ms)
2025-02-17 08:28:35.680 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822793499 ms)
2025-02-17 08:28:35.684 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:28:25.400183+00:00'} (T+822793504 ms)
2025-02-17 08:29:10.628 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822828447 ms)
2025-02-17 08:29:10.628 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822828448 ms)
2025-02-17 08:29:10.629 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41572)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822828448 ms)
2025-02-17 08:29:25.645 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822843465 ms)
2025-02-17 08:29:25.646 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822843465 ms)
2025-02-17 08:29:25.649 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:29:25.402833+00:00', 'runtime': '822840.003', 'adc_vb_f': '12.21', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822843468 ms)
2025-02-17 08:29:25.650 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:28:25.402800+00:00', 'runtime': '822780.003', 'adc_vb_f': '12.21', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822843469 ms)
2025-02-17 08:29:35.688 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822853507 ms)
2025-02-17 08:29:35.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822853508 ms)
2025-02-17 08:29:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:29:25.400196+00:00'} (T+822853513 ms)
2025-02-17 08:30:03.771 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822881590 ms)
2025-02-17 08:30:03.771 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822881591 ms)
2025-02-17 08:30:03.772 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41630)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822881591 ms)
2025-02-17 08:30:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822904421 ms)
2025-02-17 08:30:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822904421 ms)
2025-02-17 08:30:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:30:25.402796+00:00', 'runtime': '822900.003', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822904425 ms)
2025-02-17 08:30:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:29:25.402833+00:00', 'runtime': '822840.003', 'adc_vb_f': '12.21', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822904425 ms)
2025-02-17 08:30:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822913428 ms)
2025-02-17 08:30:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+822913429 ms)
2025-02-17 08:30:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:30:25.400189+00:00'} (T+822913436 ms)
2025-02-17 08:31:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+822964420 ms)
2025-02-17 08:31:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+822964421 ms)
2025-02-17 08:31:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:31:25.400158+00:00', 'runtime': '822960.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.428', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822964425 ms)
2025-02-17 08:31:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:30:25.402796+00:00', 'runtime': '822900.003', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+822964425 ms)
2025-02-17 08:32:01.981 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+822999800 ms)
2025-02-17 08:32:01.981 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+822999800 ms)
2025-02-17 08:32:01.981 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41758)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+822999801 ms)
2025-02-17 08:32:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823024409 ms)
2025-02-17 08:32:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823024409 ms)
2025-02-17 08:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:32:25.400795+00:00', 'runtime': '823020.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.431', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823024413 ms)
2025-02-17 08:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:31:25.400158+00:00', 'runtime': '822960.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.428', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823024413 ms)
2025-02-17 08:33:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823084416 ms)
2025-02-17 08:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823084416 ms)
2025-02-17 08:33:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:33:25.400970+00:00', 'runtime': '823080.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.453', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823084421 ms)
2025-02-17 08:33:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:32:25.400795+00:00', 'runtime': '823020.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.431', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823084421 ms)
2025-02-17 08:34:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823144412 ms)
2025-02-17 08:34:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823144413 ms)
2025-02-17 08:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:34:25.402857+00:00', 'runtime': '823140.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.456', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823144417 ms)
2025-02-17 08:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:33:25.400970+00:00', 'runtime': '823080.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.453', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823144417 ms)
2025-02-17 08:34:35.689 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823153508 ms)
2025-02-17 08:34:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823153509 ms)
2025-02-17 08:34:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:34:25.400191+00:00'} (T+823153513 ms)
2025-02-17 08:35:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823204414 ms)
2025-02-17 08:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823204414 ms)
2025-02-17 08:35:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:35:25.400100+00:00', 'runtime': '823200.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.443', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823204419 ms)
2025-02-17 08:35:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:34:25.402857+00:00', 'runtime': '823140.003', 'adc_vb_f': '12.204', 'adc_va_f': '1.456', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823204419 ms)
2025-02-17 08:36:00.276 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+823238096 ms)
2025-02-17 08:36:00.277 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+823238096 ms)
2025-02-17 08:36:00.277 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823238096 ms)
2025-02-17 08:36:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823264417 ms)
2025-02-17 08:36:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823264418 ms)
2025-02-17 08:36:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:36:25.400828+00:00', 'runtime': '823260.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823264421 ms)
2025-02-17 08:36:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:35:25.400100+00:00', 'runtime': '823200.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.443', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823264422 ms)
2025-02-17 08:37:09.433 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823307252 ms)
2025-02-17 08:37:09.433 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823307253 ms)
2025-02-17 08:37:09.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42090)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823307253 ms)
2025-02-17 08:37:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823324405 ms)
2025-02-17 08:37:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823324406 ms)
2025-02-17 08:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:37:25.400617+00:00', 'runtime': '823320.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823324409 ms)
2025-02-17 08:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:36:25.400828+00:00', 'runtime': '823260.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.419', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823324410 ms)
2025-02-17 08:37:57.553 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823355372 ms)
2025-02-17 08:37:57.553 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823355373 ms)
2025-02-17 08:37:57.554 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42142)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823355373 ms)
2025-02-17 08:38:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823384422 ms)
2025-02-17 08:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823384422 ms)
2025-02-17 08:38:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:38:25.415607+00:00', 'runtime': '823380.016', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823384423 ms)
2025-02-17 08:38:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:37:25.400617+00:00', 'runtime': '823320.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.002', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823384424 ms)
2025-02-17 08:38:35.644 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823393464 ms)
2025-02-17 08:38:35.645 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823393464 ms)
2025-02-17 08:38:35.646 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:38:25.413012+00:00'} (T+823393466 ms)
2025-02-17 08:39:14.712 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823432531 ms)
2025-02-17 08:39:14.712 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823432532 ms)
2025-02-17 08:39:14.713 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42224)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823432532 ms)
2025-02-17 08:39:22.724 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+823440543 ms)
2025-02-17 08:39:22.724 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+823440543 ms)
2025-02-17 08:39:22.724 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823440544 ms)
2025-02-17 08:39:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823444405 ms)
2025-02-17 08:39:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823444406 ms)
2025-02-17 08:39:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:39:25.400167+00:00', 'runtime': '823440.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823444407 ms)
2025-02-17 08:39:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:38:25.415607+00:00', 'runtime': '823380.016', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823444407 ms)
2025-02-17 08:39:35.838 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823453658 ms)
2025-02-17 08:39:35.839 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823453658 ms)
2025-02-17 08:39:35.846 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:39:25.400156+00:00'} (T+823453665 ms)
2025-02-17 08:40:11.871 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823489691 ms)
2025-02-17 08:40:11.872 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823489691 ms)
2025-02-17 08:40:11.872 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42286)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823489691 ms)
2025-02-17 08:40:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823504407 ms)
2025-02-17 08:40:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823504408 ms)
2025-02-17 08:40:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:40:25.400172+00:00', 'runtime': '823500.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823504409 ms)
2025-02-17 08:40:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:39:25.400167+00:00', 'runtime': '823440.001', 'adc_vb_f': '12.207', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823504409 ms)
2025-02-17 08:41:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823564406 ms)
2025-02-17 08:41:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823564407 ms)
2025-02-17 08:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:41:25.400170+00:00', 'runtime': '823560.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823564408 ms)
2025-02-17 08:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:40:25.400172+00:00', 'runtime': '823500.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.1', 'ahl_t': '1588.1', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823564408 ms)
2025-02-17 08:42:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823624408 ms)
2025-02-17 08:42:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823624408 ms)
2025-02-17 08:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:42:25.400271+00:00', 'runtime': '823620.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823624409 ms)
2025-02-17 08:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:41:25.400170+00:00', 'runtime': '823560.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823624410 ms)
2025-02-17 08:43:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823684409 ms)
2025-02-17 08:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823684409 ms)
2025-02-17 08:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:43:25.400187+00:00', 'runtime': '823680.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823684410 ms)
2025-02-17 08:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:42:25.400271+00:00', 'runtime': '823620.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823684411 ms)
2025-02-17 08:43:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823693439 ms)
2025-02-17 08:43:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823693440 ms)
2025-02-17 08:43:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:43:25.400342+00:00'} (T+823693441 ms)
2025-02-17 08:44:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823744409 ms)
2025-02-17 08:44:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823744409 ms)
2025-02-17 08:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:44:25.400216+00:00', 'runtime': '823740.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.425', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823744410 ms)
2025-02-17 08:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:43:25.400187+00:00', 'runtime': '823680.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823744411 ms)
2025-02-17 08:44:34.241 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823752060 ms)
2025-02-17 08:44:34.241 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823752060 ms)
2025-02-17 08:44:34.242 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42564)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823752061 ms)
2025-02-17 08:44:35.712 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823753531 ms)
2025-02-17 08:44:35.713 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+823753532 ms)
2025-02-17 08:44:35.715 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:44:25.400221+00:00'} (T+823753534 ms)
2025-02-17 08:45:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823804408 ms)
2025-02-17 08:45:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823804409 ms)
2025-02-17 08:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:45:25.400169+00:00', 'runtime': '823800.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.422', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823804410 ms)
2025-02-17 08:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:44:25.400216+00:00', 'runtime': '823740.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.425', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823804410 ms)
2025-02-17 08:46:15.451 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823853270 ms)
2025-02-17 08:46:15.451 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823853271 ms)
2025-02-17 08:46:15.452 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42682)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823853271 ms)
2025-02-17 08:46:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823864410 ms)
2025-02-17 08:46:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823864410 ms)
2025-02-17 08:46:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:46:25.400170+00:00', 'runtime': '823860.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823864411 ms)
2025-02-17 08:46:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:45:25.400169+00:00', 'runtime': '823800.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.422', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.198', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823864412 ms)
2025-02-17 08:46:43.553 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823881372 ms)
2025-02-17 08:46:43.553 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823881372 ms)
2025-02-17 08:46:43.553 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42712)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823881372 ms)
2025-02-17 08:47:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823924411 ms)
2025-02-17 08:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823924411 ms)
2025-02-17 08:47:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:47:25.400200+00:00', 'runtime': '823920.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823924412 ms)
2025-02-17 08:47:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:46:25.400170+00:00', 'runtime': '823860.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823924413 ms)
2025-02-17 08:47:46.724 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+823944543 ms)
2025-02-17 08:47:46.724 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+823944544 ms)
2025-02-17 08:47:46.725 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42780)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+823944544 ms)
2025-02-17 08:48:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+823984413 ms)
2025-02-17 08:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+823984414 ms)
2025-02-17 08:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:48:25.400170+00:00', 'runtime': '823980.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.407', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823984415 ms)
2025-02-17 08:48:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:47:25.400200+00:00', 'runtime': '823920.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+823984415 ms)
2025-02-17 08:49:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824044414 ms)
2025-02-17 08:49:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824044414 ms)
2025-02-17 08:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:49:25.400172+00:00', 'runtime': '824040.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.392', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824044415 ms)
2025-02-17 08:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:48:25.400170+00:00', 'runtime': '823980.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.407', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824044415 ms)
2025-02-17 08:50:25.993 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824103812 ms)
2025-02-17 08:50:25.993 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824103812 ms)
2025-02-17 08:50:25.993 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42946)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824103812 ms)
2025-02-17 08:50:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824104415 ms)
2025-02-17 08:50:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824104416 ms)
2025-02-17 08:50:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:50:25.400162+00:00', 'runtime': '824100.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.392', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824104417 ms)
2025-02-17 08:50:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:49:25.400172+00:00', 'runtime': '824040.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.392', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824104417 ms)
2025-02-17 08:50:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824113422 ms)
2025-02-17 08:50:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824113423 ms)
2025-02-17 08:50:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:50:25.400347+00:00'} (T+824113428 ms)
2025-02-17 08:50:56.092 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824133912 ms)
2025-02-17 08:50:56.093 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824133912 ms)
2025-02-17 08:50:56.093 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42982)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824133912 ms)
2025-02-17 08:51:15.212 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824153031 ms)
2025-02-17 08:51:15.212 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824153031 ms)
2025-02-17 08:51:15.212 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43002)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824153032 ms)
2025-02-17 08:51:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824164417 ms)
2025-02-17 08:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824164417 ms)
2025-02-17 08:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:51:25.400197+00:00', 'runtime': '824160.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.398', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.192', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824164418 ms)
2025-02-17 08:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:50:25.400162+00:00', 'runtime': '824100.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.392', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.195', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824164418 ms)
2025-02-17 08:51:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824173425 ms)
2025-02-17 08:51:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824173426 ms)
2025-02-17 08:51:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:51:25.400220+00:00'} (T+824173433 ms)
2025-02-17 08:51:47.253 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824185073 ms)
2025-02-17 08:51:47.254 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824185073 ms)
2025-02-17 08:51:47.254 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824185074 ms)
2025-02-17 08:51:47.255 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824185075 ms)
2025-02-17 08:51:47.256 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824185075 ms)
2025-02-17 08:51:47.256 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824185076 ms)
2025-02-17 08:51:47.258 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824185077 ms)
2025-02-17 08:51:47.258 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824185077 ms)
2025-02-17 08:51:47.259 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824185078 ms)
2025-02-17 08:51:47.260 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824185079 ms)
2025-02-17 08:51:47.260 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824185079 ms)
2025-02-17 08:51:47.261 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824185080 ms)
2025-02-17 08:51:47.262 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824185081 ms)
2025-02-17 08:51:47.262 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824185081 ms)
2025-02-17 08:51:47.263 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824185082 ms)
2025-02-17 08:51:47.264 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824185083 ms)
2025-02-17 08:51:47.264 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824185083 ms)
2025-02-17 08:51:47.266 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824185085 ms)
2025-02-17 08:52:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824224416 ms)
2025-02-17 08:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824224416 ms)
2025-02-17 08:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:52:25.400174+00:00', 'runtime': '824220.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.413', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.192', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824224417 ms)
2025-02-17 08:52:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:51:25.400197+00:00', 'runtime': '824160.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.398', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.192', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824224418 ms)
2025-02-17 08:52:35.814 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824233633 ms)
2025-02-17 08:52:35.815 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824233634 ms)
2025-02-17 08:52:35.817 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:52:25.400201+00:00'} (T+824233636 ms)
2025-02-17 08:53:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824284419 ms)
2025-02-17 08:53:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824284420 ms)
2025-02-17 08:53:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:53:25.400165+00:00', 'runtime': '824280.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.413', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.192', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824284421 ms)
2025-02-17 08:53:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:52:25.400174+00:00', 'runtime': '824220.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.413', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.192', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824284421 ms)
2025-02-17 08:53:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824293428 ms)
2025-02-17 08:53:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824293428 ms)
2025-02-17 08:53:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:53:25.400011+00:00'} (T+824293430 ms)
2025-02-17 08:54:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824344419 ms)
2025-02-17 08:54:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824344419 ms)
2025-02-17 08:54:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:54:25.400285+00:00', 'runtime': '824340.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.425', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.189', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824344420 ms)
2025-02-17 08:54:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:53:25.400165+00:00', 'runtime': '824280.001', 'adc_vb_f': '12.204', 'adc_va_f': '1.413', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.2', 'ahl_t': '1588.2', 'hourmeter': '20115', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.192', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824344421 ms)
2025-02-17 08:55:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824404404 ms)
2025-02-17 08:55:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824404405 ms)
2025-02-17 08:55:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:55:25.400157+00:00', 'runtime': '824400.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.44', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.189', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824404406 ms)
2025-02-17 08:55:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:54:25.400285+00:00', 'runtime': '824340.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.425', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.189', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824404406 ms)
2025-02-17 08:55:42.661 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824420480 ms)
2025-02-17 08:55:42.661 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824420481 ms)
2025-02-17 08:55:42.662 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43286)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824420481 ms)
2025-02-17 08:55:55.772 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824433592 ms)
2025-02-17 08:55:55.773 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824433592 ms)
2025-02-17 08:55:55.773 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43312)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824433592 ms)
2025-02-17 08:56:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824464407 ms)
2025-02-17 08:56:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824464408 ms)
2025-02-17 08:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:56:25.400169+00:00', 'runtime': '824460.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.419', 'adc_vl_f': '12.195', 'adc_ic_f': '0.002', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.189', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824464409 ms)
2025-02-17 08:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:55:25.400157+00:00', 'runtime': '824400.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.44', 'adc_vl_f': '12.204', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.189', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824464409 ms)
2025-02-17 08:56:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824473430 ms)
2025-02-17 08:56:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824473431 ms)
2025-02-17 08:56:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:56:25.400332+00:00'} (T+824473433 ms)
2025-02-17 08:56:57.913 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824495733 ms)
2025-02-17 08:56:57.914 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824495733 ms)
2025-02-17 08:56:57.914 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43380)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824495733 ms)
2025-02-17 08:57:10.002 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824507821 ms)
2025-02-17 08:57:10.002 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824507822 ms)
2025-02-17 08:57:10.003 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43394)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824507822 ms)
2025-02-17 08:57:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824524406 ms)
2025-02-17 08:57:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824524406 ms)
2025-02-17 08:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:57:25.400172+00:00', 'runtime': '824520.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824524410 ms)
2025-02-17 08:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:56:25.400169+00:00', 'runtime': '824460.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.419', 'adc_vl_f': '12.195', 'adc_ic_f': '0.002', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.189', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824524410 ms)
2025-02-17 08:57:33.485 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+824531304 ms)
2025-02-17 08:58:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824584410 ms)
2025-02-17 08:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824584410 ms)
2025-02-17 08:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:58:25.400185+00:00', 'runtime': '824580.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.413', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824584414 ms)
2025-02-17 08:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:57:25.400172+00:00', 'runtime': '824520.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.413', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824584415 ms)
2025-02-17 08:58:58.216 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824616035 ms)
2025-02-17 08:58:58.216 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824616035 ms)
2025-02-17 08:58:58.217 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43508)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824616036 ms)
2025-02-17 08:59:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824644411 ms)
2025-02-17 08:59:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824644411 ms)
2025-02-17 08:59:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 08:59:25.400202+00:00', 'runtime': '824640.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824644414 ms)
2025-02-17 08:59:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:58:25.400185+00:00', 'runtime': '824580.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.413', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824644415 ms)
2025-02-17 08:59:35.787 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824653606 ms)
2025-02-17 08:59:35.788 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824653607 ms)
2025-02-17 08:59:35.789 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 08:59:25.400340+00:00'} (T+824653608 ms)
2025-02-17 08:59:54.359 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824672178 ms)
2025-02-17 08:59:54.359 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824672178 ms)
2025-02-17 08:59:54.359 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43570)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824672179 ms)
2025-02-17 09:00:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824704412 ms)
2025-02-17 09:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824704413 ms)
2025-02-17 09:00:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:00:25.400201+00:00', 'runtime': '824700.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.422', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824704417 ms)
2025-02-17 09:00:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 08:59:25.400202+00:00', 'runtime': '824640.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.416', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824704417 ms)
2025-02-17 09:00:35.727 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824713546 ms)
2025-02-17 09:00:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824713547 ms)
2025-02-17 09:00:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:00:25.400007+00:00'} (T+824713548 ms)
2025-02-17 09:00:52.429 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824730249 ms)
2025-02-17 09:00:52.430 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824730249 ms)
2025-02-17 09:00:52.430 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824730250 ms)
2025-02-17 09:00:52.431 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824730251 ms)
2025-02-17 09:00:52.432 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824730251 ms)
2025-02-17 09:00:52.432 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824730251 ms)
2025-02-17 09:00:52.433 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824730253 ms)
2025-02-17 09:00:52.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824730253 ms)
2025-02-17 09:00:52.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824730254 ms)
2025-02-17 09:00:52.435 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+824730255 ms)
2025-02-17 09:00:52.436 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+824730255 ms)
2025-02-17 09:00:52.436 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824730256 ms)
2025-02-17 09:01:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824764415 ms)
2025-02-17 09:01:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824764416 ms)
2025-02-17 09:01:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:01:25.400205+00:00', 'runtime': '824760.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.45', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824764418 ms)
2025-02-17 09:01:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:00:25.400201+00:00', 'runtime': '824700.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.422', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824764418 ms)
2025-02-17 09:01:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824773428 ms)
2025-02-17 09:01:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+824773429 ms)
2025-02-17 09:01:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:01:25.400124+00:00'} (T+824773430 ms)
2025-02-17 09:02:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824824418 ms)
2025-02-17 09:02:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824824418 ms)
2025-02-17 09:02:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:02:25.400194+00:00', 'runtime': '824820.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.456', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824824423 ms)
2025-02-17 09:02:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:01:25.400205+00:00', 'runtime': '824760.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.45', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.186', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824824423 ms)
2025-02-17 09:03:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824884404 ms)
2025-02-17 09:03:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824884405 ms)
2025-02-17 09:03:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:03:25.400609+00:00', 'runtime': '824880.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.456', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824884409 ms)
2025-02-17 09:03:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:02:25.400194+00:00', 'runtime': '824820.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.456', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824884409 ms)
2025-02-17 09:04:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+824944409 ms)
2025-02-17 09:04:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+824944410 ms)
2025-02-17 09:04:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:04:25.400623+00:00', 'runtime': '824940.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.459', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824944414 ms)
2025-02-17 09:04:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:03:25.400609+00:00', 'runtime': '824880.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.456', 'adc_vl_f': '12.201', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+824944414 ms)
2025-02-17 09:04:41.789 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+824959608 ms)
2025-02-17 09:04:41.789 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+824959609 ms)
2025-02-17 09:04:41.790 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43872)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+824959609 ms)
2025-02-17 09:05:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825004412 ms)
2025-02-17 09:05:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825004412 ms)
2025-02-17 09:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:05:25.402809+00:00', 'runtime': '825000.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825004416 ms)
2025-02-17 09:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:04:25.400623+00:00', 'runtime': '824940.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.459', 'adc_vl_f': '12.192', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825004417 ms)
2025-02-17 09:06:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825064416 ms)
2025-02-17 09:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825064416 ms)
2025-02-17 09:06:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:06:25.400374+00:00', 'runtime': '825060.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.468', 'adc_vl_f': '12.189', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825064419 ms)
2025-02-17 09:06:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:05:25.402809+00:00', 'runtime': '825000.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.183', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825064420 ms)
2025-02-17 09:07:08.049 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825105868 ms)
2025-02-17 09:07:08.049 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825105868 ms)
2025-02-17 09:07:08.050 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44038)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825105869 ms)
2025-02-17 09:07:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825124414 ms)
2025-02-17 09:07:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825124415 ms)
2025-02-17 09:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:07:25.400166+00:00', 'runtime': '825120.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.18', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825124419 ms)
2025-02-17 09:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:06:25.400374+00:00', 'runtime': '825060.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.468', 'adc_vl_f': '12.189', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.3', 'ahl_t': '1588.3', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825124419 ms)
2025-02-17 09:07:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825133438 ms)
2025-02-17 09:07:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825133438 ms)
2025-02-17 09:07:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:07:25.405062+00:00'} (T+825133440 ms)
2025-02-17 09:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825184414 ms)
2025-02-17 09:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825184414 ms)
2025-02-17 09:08:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:08:25.402863+00:00', 'runtime': '825180.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.48', 'adc_vl_f': '12.18', 'adc_ic_f': '0.002', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825184419 ms)
2025-02-17 09:08:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:07:25.400166+00:00', 'runtime': '825120.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.18', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825184419 ms)
2025-02-17 09:08:35.667 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825193486 ms)
2025-02-17 09:08:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825193487 ms)
2025-02-17 09:08:35.668 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:08:25.400226+00:00'} (T+825193488 ms)
2025-02-17 09:09:18.282 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825236101 ms)
2025-02-17 09:09:18.282 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825236101 ms)
2025-02-17 09:09:18.283 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44178)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825236102 ms)
2025-02-17 09:09:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825244419 ms)
2025-02-17 09:09:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825244420 ms)
2025-02-17 09:09:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:09:25.402773+00:00', 'runtime': '825240.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.483', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825244423 ms)
2025-02-17 09:09:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:08:25.402863+00:00', 'runtime': '825180.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.48', 'adc_vl_f': '12.18', 'adc_ic_f': '0.002', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825244424 ms)
2025-02-17 09:10:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825304404 ms)
2025-02-17 09:10:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825304405 ms)
2025-02-17 09:10:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:10:25.400732+00:00', 'runtime': '825300.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825304409 ms)
2025-02-17 09:10:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:09:25.402773+00:00', 'runtime': '825240.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.483', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825304409 ms)
2025-02-17 09:10:34.438 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825312257 ms)
2025-02-17 09:10:34.438 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825312257 ms)
2025-02-17 09:10:34.438 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44260)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825312258 ms)
2025-02-17 09:11:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825364421 ms)
2025-02-17 09:11:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825364421 ms)
2025-02-17 09:11:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:11:25.400170+00:00', 'runtime': '825360.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825364425 ms)
2025-02-17 09:11:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:10:25.400732+00:00', 'runtime': '825300.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825364426 ms)
2025-02-17 09:11:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825373440 ms)
2025-02-17 09:11:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825373440 ms)
2025-02-17 09:11:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:11:25.413050+00:00'} (T+825373442 ms)
2025-02-17 09:12:01.630 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825399449 ms)
2025-02-17 09:12:01.630 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825399449 ms)
2025-02-17 09:12:01.630 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44354)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825399450 ms)
2025-02-17 09:12:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825424407 ms)
2025-02-17 09:12:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825424408 ms)
2025-02-17 09:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:12:25.402789+00:00', 'runtime': '825420.003', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825424411 ms)
2025-02-17 09:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:11:25.400170+00:00', 'runtime': '825360.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.18', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825424412 ms)
2025-02-17 09:12:35.738 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825433558 ms)
2025-02-17 09:12:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825433558 ms)
2025-02-17 09:12:35.740 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:12:25.400153+00:00'} (T+825433559 ms)
2025-02-17 09:13:09.769 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825467588 ms)
2025-02-17 09:13:09.769 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825467589 ms)
2025-02-17 09:13:09.770 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44428)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825467589 ms)
2025-02-17 09:13:13.851 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825471670 ms)
2025-02-17 09:13:13.851 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825471670 ms)
2025-02-17 09:13:13.851 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44434)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825471671 ms)
2025-02-17 09:13:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825484410 ms)
2025-02-17 09:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825484410 ms)
2025-02-17 09:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:13:25.402701+00:00', 'runtime': '825480.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.474', 'adc_vl_f': '12.177', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825484414 ms)
2025-02-17 09:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:12:25.402789+00:00', 'runtime': '825420.003', 'adc_vb_f': '12.201', 'adc_va_f': '1.48', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825484414 ms)
2025-02-17 09:13:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825493430 ms)
2025-02-17 09:13:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825493431 ms)
2025-02-17 09:13:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:13:25.400065+00:00'} (T+825493432 ms)
2025-02-17 09:13:52.982 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825510802 ms)
2025-02-17 09:13:52.982 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825510802 ms)
2025-02-17 09:13:52.983 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44478)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825510802 ms)
2025-02-17 09:14:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825544409 ms)
2025-02-17 09:14:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825544409 ms)
2025-02-17 09:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:14:25.402868+00:00', 'runtime': '825540.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.471', 'adc_vl_f': '12.17', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825544413 ms)
2025-02-17 09:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:13:25.402701+00:00', 'runtime': '825480.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.474', 'adc_vl_f': '12.177', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825544414 ms)
2025-02-17 09:14:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825553432 ms)
2025-02-17 09:14:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825553433 ms)
2025-02-17 09:14:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:14:25.400269+00:00'} (T+825553434 ms)
2025-02-17 09:15:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825604410 ms)
2025-02-17 09:15:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825604410 ms)
2025-02-17 09:15:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:15:25.402844+00:00', 'runtime': '825600.003', 'adc_vb_f': '12.201', 'adc_va_f': '1.465', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825604415 ms)
2025-02-17 09:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:14:25.402868+00:00', 'runtime': '825540.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.471', 'adc_vl_f': '12.17', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825604415 ms)
2025-02-17 09:15:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825613437 ms)
2025-02-17 09:15:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825613438 ms)
2025-02-17 09:15:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:15:25.400239+00:00'} (T+825613439 ms)
2025-02-17 09:16:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825664410 ms)
2025-02-17 09:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825664410 ms)
2025-02-17 09:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:16:25.400200+00:00', 'runtime': '825660.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.177', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825664414 ms)
2025-02-17 09:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:15:25.402844+00:00', 'runtime': '825600.003', 'adc_vb_f': '12.201', 'adc_va_f': '1.465', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825664414 ms)
2025-02-17 09:16:35.651 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825673470 ms)
2025-02-17 09:16:35.652 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825673471 ms)
2025-02-17 09:16:35.653 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:16:25.400211+00:00'} (T+825673472 ms)
2025-02-17 09:17:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825724412 ms)
2025-02-17 09:17:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825724412 ms)
2025-02-17 09:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:17:25.402805+00:00', 'runtime': '825720.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.453', 'adc_vl_f': '12.17', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825724416 ms)
2025-02-17 09:17:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:16:25.400200+00:00', 'runtime': '825660.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.177', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825724417 ms)
2025-02-17 09:17:31.311 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+825729131 ms)
2025-02-17 09:17:31.312 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+825729131 ms)
2025-02-17 09:17:31.312 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44720)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+825729131 ms)
2025-02-17 09:17:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825733434 ms)
2025-02-17 09:17:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825733434 ms)
2025-02-17 09:17:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:17:25.400170+00:00'} (T+825733435 ms)
2025-02-17 09:18:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825784414 ms)
2025-02-17 09:18:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825784415 ms)
2025-02-17 09:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:18:25.402794+00:00', 'runtime': '825780.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.183', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825784419 ms)
2025-02-17 09:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:17:25.402805+00:00', 'runtime': '825720.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.453', 'adc_vl_f': '12.17', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825784419 ms)
2025-02-17 09:19:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825844415 ms)
2025-02-17 09:19:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825844415 ms)
2025-02-17 09:19:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:19:25.400070+00:00', 'runtime': '825840.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.459', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825844420 ms)
2025-02-17 09:19:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:18:25.402794+00:00', 'runtime': '825780.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.183', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825844420 ms)
2025-02-17 09:20:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825904415 ms)
2025-02-17 09:20:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825904416 ms)
2025-02-17 09:20:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:20:25.400588+00:00', 'runtime': '825900.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825904420 ms)
2025-02-17 09:20:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:19:25.400070+00:00', 'runtime': '825840.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.459', 'adc_vl_f': '12.198', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.4', 'ahl_t': '1588.4', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825904420 ms)
2025-02-17 09:21:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+825964415 ms)
2025-02-17 09:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+825964415 ms)
2025-02-17 09:21:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:21:25.400191+00:00', 'runtime': '825960.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.459', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825964420 ms)
2025-02-17 09:21:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:20:25.400588+00:00', 'runtime': '825900.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.195', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+825964420 ms)
2025-02-17 09:21:35.624 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825973444 ms)
2025-02-17 09:21:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+825973444 ms)
2025-02-17 09:21:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:21:25.413048+00:00'} (T+825973445 ms)
2025-02-17 09:22:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826024419 ms)
2025-02-17 09:22:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826024420 ms)
2025-02-17 09:22:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:22:25.402806+00:00', 'runtime': '826020.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.177', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826024424 ms)
2025-02-17 09:22:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:21:25.400191+00:00', 'runtime': '825960.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.459', 'adc_vl_f': '12.186', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826024424 ms)
2025-02-17 09:22:35.751 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826033570 ms)
2025-02-17 09:22:35.752 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826033571 ms)
2025-02-17 09:22:35.752 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:22:25.400180+00:00'} (T+826033572 ms)
2025-02-17 09:23:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826084421 ms)
2025-02-17 09:23:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826084422 ms)
2025-02-17 09:23:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:23:25.402638+00:00', 'runtime': '826080.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.183', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826084426 ms)
2025-02-17 09:23:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:22:25.402806+00:00', 'runtime': '826020.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.177', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826084426 ms)
2025-02-17 09:23:35.678 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826093497 ms)
2025-02-17 09:23:35.679 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826093498 ms)
2025-02-17 09:23:35.680 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:23:25.400042+00:00'} (T+826093499 ms)
2025-02-17 09:23:59.855 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826117674 ms)
2025-02-17 09:23:59.855 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826117674 ms)
2025-02-17 09:23:59.855 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45134)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826117675 ms)
2025-02-17 09:24:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826144406 ms)
2025-02-17 09:24:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826144406 ms)
2025-02-17 09:24:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:24:25.402747+00:00', 'runtime': '826140.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826144410 ms)
2025-02-17 09:24:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:23:25.402638+00:00', 'runtime': '826080.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.183', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826144410 ms)
2025-02-17 09:24:35.686 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826153505 ms)
2025-02-17 09:24:35.687 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826153506 ms)
2025-02-17 09:24:35.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:24:25.400113+00:00'} (T+826153507 ms)
2025-02-17 09:25:06.004 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826183824 ms)
2025-02-17 09:25:06.005 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826183824 ms)
2025-02-17 09:25:06.005 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45206)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826183824 ms)
2025-02-17 09:25:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826204409 ms)
2025-02-17 09:25:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826204410 ms)
2025-02-17 09:25:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:25:25.400208+00:00', 'runtime': '826200.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.456', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826204411 ms)
2025-02-17 09:25:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:24:25.402747+00:00', 'runtime': '826140.003', 'adc_vb_f': '12.198', 'adc_va_f': '1.456', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.198', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826204411 ms)
2025-02-17 09:26:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826264411 ms)
2025-02-17 09:26:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826264412 ms)
2025-02-17 09:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:26:25.400183+00:00', 'runtime': '826260.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.456', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826264413 ms)
2025-02-17 09:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:25:25.400208+00:00', 'runtime': '826200.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.456', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826264416 ms)
2025-02-17 09:26:55.234 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826293053 ms)
2025-02-17 09:26:55.234 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826293053 ms)
2025-02-17 09:26:55.235 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45332)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826293054 ms)
2025-02-17 09:26:59.306 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826297125 ms)
2025-02-17 09:26:59.306 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826297126 ms)
2025-02-17 09:26:59.307 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45338)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826297126 ms)
2025-02-17 09:27:26.609 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826324428 ms)
2025-02-17 09:27:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826324429 ms)
2025-02-17 09:27:26.611 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:27:25.400173+00:00', 'runtime': '826320.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.456', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826324430 ms)
2025-02-17 09:27:26.613 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:26:25.400183+00:00', 'runtime': '826260.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.456', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826324432 ms)
2025-02-17 09:27:35.794 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826333613 ms)
2025-02-17 09:27:35.795 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826333614 ms)
2025-02-17 09:27:35.796 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:27:25.400330+00:00'} (T+826333615 ms)
2025-02-17 09:28:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826384414 ms)
2025-02-17 09:28:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826384415 ms)
2025-02-17 09:28:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:28:25.400153+00:00', 'runtime': '826380.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.459', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826384416 ms)
2025-02-17 09:28:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:27:25.400173+00:00', 'runtime': '826320.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.456', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826384416 ms)
2025-02-17 09:29:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826444416 ms)
2025-02-17 09:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826444416 ms)
2025-02-17 09:29:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:29:25.400166+00:00', 'runtime': '826440.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.462', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826444417 ms)
2025-02-17 09:29:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:28:25.400153+00:00', 'runtime': '826380.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.459', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826444418 ms)
2025-02-17 09:30:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826504415 ms)
2025-02-17 09:30:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826504415 ms)
2025-02-17 09:30:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:30:25.400169+00:00', 'runtime': '826500.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.474', 'adc_vl_f': '12.183', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826504416 ms)
2025-02-17 09:30:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:29:25.400166+00:00', 'runtime': '826440.001', 'adc_vb_f': '12.201', 'adc_va_f': '1.462', 'adc_vl_f': '12.173', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826504416 ms)
2025-02-17 09:30:33.574 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+826511393 ms)
2025-02-17 09:30:33.574 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+826511393 ms)
2025-02-17 09:30:33.575 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826511395 ms)
2025-02-17 09:30:35.775 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826513594 ms)
2025-02-17 09:30:35.776 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826513595 ms)
2025-02-17 09:30:35.777 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:30:25.400689+00:00'} (T+826513596 ms)
2025-02-17 09:31:06.706 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826544525 ms)
2025-02-17 09:31:06.706 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826544525 ms)
2025-02-17 09:31:06.707 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45600)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826544526 ms)
2025-02-17 09:31:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826564415 ms)
2025-02-17 09:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826564415 ms)
2025-02-17 09:31:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:31:25.400173+00:00', 'runtime': '826560.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.474', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826564416 ms)
2025-02-17 09:31:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:30:25.400169+00:00', 'runtime': '826500.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.474', 'adc_vl_f': '12.183', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826564418 ms)
2025-02-17 09:31:35.595 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826573414 ms)
2025-02-17 09:31:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826573415 ms)
2025-02-17 09:31:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:31:25.400000+00:00'} (T+826573415 ms)
2025-02-17 09:31:38.814 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826576633 ms)
2025-02-17 09:31:38.814 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826576633 ms)
2025-02-17 09:31:38.815 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45636)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826576634 ms)
2025-02-17 09:31:53.932 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826591751 ms)
2025-02-17 09:31:53.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826591751 ms)
2025-02-17 09:31:53.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45652)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826591752 ms)
2025-02-17 09:32:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826624414 ms)
2025-02-17 09:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826624415 ms)
2025-02-17 09:32:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:32:25.400171+00:00', 'runtime': '826620.001', 'adc_vb_f': '12.186', 'adc_va_f': '1.477', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.507', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826624416 ms)
2025-02-17 09:32:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:31:25.400173+00:00', 'runtime': '826560.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.474', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.5', 'ahl_t': '1588.5', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826624416 ms)
2025-02-17 09:32:35.726 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826633546 ms)
2025-02-17 09:32:35.727 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826633546 ms)
2025-02-17 09:32:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:32:25.400219+00:00'} (T+826633547 ms)
2025-02-17 09:33:22.118 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826679938 ms)
2025-02-17 09:33:22.119 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826679938 ms)
2025-02-17 09:33:22.119 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45748)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826679938 ms)
2025-02-17 09:33:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826684414 ms)
2025-02-17 09:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826684414 ms)
2025-02-17 09:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:33:25.400162+00:00', 'runtime': '826680.001', 'adc_vb_f': '12.192', 'adc_va_f': '1.477', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826684415 ms)
2025-02-17 09:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:32:25.400171+00:00', 'runtime': '826620.001', 'adc_vb_f': '12.186', 'adc_va_f': '1.477', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.507', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826684416 ms)
2025-02-17 09:33:35.637 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826693456 ms)
2025-02-17 09:33:35.638 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826693457 ms)
2025-02-17 09:33:35.639 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:33:25.400049+00:00'} (T+826693458 ms)
2025-02-17 09:34:05.257 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826723076 ms)
2025-02-17 09:34:05.257 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826723077 ms)
2025-02-17 09:34:05.258 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45796)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826723077 ms)
2025-02-17 09:34:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826744416 ms)
2025-02-17 09:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826744416 ms)
2025-02-17 09:34:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:34:25.400254+00:00', 'runtime': '826740.001', 'adc_vb_f': '12.192', 'adc_va_f': '1.465', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826744418 ms)
2025-02-17 09:34:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:33:25.400162+00:00', 'runtime': '826680.001', 'adc_vb_f': '12.192', 'adc_va_f': '1.477', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826744419 ms)
2025-02-17 09:34:35.637 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826753456 ms)
2025-02-17 09:34:35.638 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826753457 ms)
2025-02-17 09:34:35.639 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:34:25.400258+00:00'} (T+826753458 ms)
2025-02-17 09:35:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826804416 ms)
2025-02-17 09:35:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826804416 ms)
2025-02-17 09:35:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:35:25.400189+00:00', 'runtime': '826800.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.462', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826804417 ms)
2025-02-17 09:35:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:34:25.400254+00:00', 'runtime': '826740.001', 'adc_vb_f': '12.192', 'adc_va_f': '1.465', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826804418 ms)
2025-02-17 09:36:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826864402 ms)
2025-02-17 09:36:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826864402 ms)
2025-02-17 09:36:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:36:25.400181+00:00', 'runtime': '826860.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.465', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826864403 ms)
2025-02-17 09:36:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:35:25.400189+00:00', 'runtime': '826800.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.462', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826864403 ms)
2025-02-17 09:37:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826924416 ms)
2025-02-17 09:37:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826924417 ms)
2025-02-17 09:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:37:25.400208+00:00', 'runtime': '826920.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.468', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826924417 ms)
2025-02-17 09:37:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:36:25.400181+00:00', 'runtime': '826860.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.465', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826924418 ms)
2025-02-17 09:37:35.771 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826933591 ms)
2025-02-17 09:37:35.772 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826933591 ms)
2025-02-17 09:37:35.773 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:37:25.400340+00:00'} (T+826933592 ms)
2025-02-17 09:38:08.634 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+826966454 ms)
2025-02-17 09:38:08.635 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+826966454 ms)
2025-02-17 09:38:08.635 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46064)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+826966454 ms)
2025-02-17 09:38:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+826984418 ms)
2025-02-17 09:38:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+826984419 ms)
2025-02-17 09:38:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:38:25.402611+00:00', 'runtime': '826980.003', 'adc_vb_f': '12.195', 'adc_va_f': '1.462', 'adc_vl_f': '12.17', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826984420 ms)
2025-02-17 09:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:37:25.400208+00:00', 'runtime': '826920.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.468', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+826984422 ms)
2025-02-17 09:38:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826993439 ms)
2025-02-17 09:38:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+826993440 ms)
2025-02-17 09:38:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:38:25.400007+00:00'} (T+826993441 ms)
2025-02-17 09:39:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827044419 ms)
2025-02-17 09:39:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827044419 ms)
2025-02-17 09:39:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:39:25.402807+00:00', 'runtime': '827040.003', 'adc_vb_f': '12.195', 'adc_va_f': '1.453', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827044423 ms)
2025-02-17 09:39:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:38:25.402611+00:00', 'runtime': '826980.003', 'adc_vb_f': '12.195', 'adc_va_f': '1.462', 'adc_vl_f': '12.17', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827044424 ms)
2025-02-17 09:39:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827053423 ms)
2025-02-17 09:39:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827053424 ms)
2025-02-17 09:39:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:39:25.400170+00:00'} (T+827053425 ms)
2025-02-17 09:40:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827104418 ms)
2025-02-17 09:40:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827104419 ms)
2025-02-17 09:40:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:40:25.400147+00:00', 'runtime': '827100.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.41', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827104423 ms)
2025-02-17 09:40:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:39:25.402807+00:00', 'runtime': '827040.003', 'adc_vb_f': '12.195', 'adc_va_f': '1.453', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827104424 ms)
2025-02-17 09:40:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827113436 ms)
2025-02-17 09:40:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827113437 ms)
2025-02-17 09:40:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:40:25.413123+00:00'} (T+827113438 ms)
2025-02-17 09:41:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827164418 ms)
2025-02-17 09:41:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827164419 ms)
2025-02-17 09:41:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:41:25.400161+00:00', 'runtime': '827160.001', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827164423 ms)
2025-02-17 09:41:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:40:25.400147+00:00', 'runtime': '827100.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.41', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827164423 ms)
2025-02-17 09:41:35.624 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827173443 ms)
2025-02-17 09:41:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827173444 ms)
2025-02-17 09:41:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:41:25.413120+00:00'} (T+827173445 ms)
2025-02-17 09:42:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827224417 ms)
2025-02-17 09:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827224418 ms)
2025-02-17 09:42:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:42:25.402806+00:00', 'runtime': '827220.003', 'adc_vb_f': '12.189', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827224422 ms)
2025-02-17 09:42:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:41:25.400161+00:00', 'runtime': '827160.001', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827224422 ms)
2025-02-17 09:42:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827233437 ms)
2025-02-17 09:42:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827233437 ms)
2025-02-17 09:42:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:42:25.400193+00:00'} (T+827233438 ms)
2025-02-17 09:42:37.031 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+827234851 ms)
2025-02-17 09:42:37.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+827234851 ms)
2025-02-17 09:42:37.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46352)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+827234852 ms)
2025-02-17 09:43:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827284418 ms)
2025-02-17 09:43:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827284418 ms)
2025-02-17 09:43:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:43:25.402796+00:00', 'runtime': '827280.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827284423 ms)
2025-02-17 09:43:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:42:25.402806+00:00', 'runtime': '827220.003', 'adc_vb_f': '12.189', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827284423 ms)
2025-02-17 09:43:35.623 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827293442 ms)
2025-02-17 09:43:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827293443 ms)
2025-02-17 09:43:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:43:25.400176+00:00'} (T+827293450 ms)
2025-02-17 09:44:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827344419 ms)
2025-02-17 09:44:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827344419 ms)
2025-02-17 09:44:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:44:25.402853+00:00', 'runtime': '827340.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827344421 ms)
2025-02-17 09:44:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:43:25.402796+00:00', 'runtime': '827280.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827344422 ms)
2025-02-17 09:44:35.680 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827353500 ms)
2025-02-17 09:44:35.681 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827353500 ms)
2025-02-17 09:44:35.682 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:44:25.400220+00:00'} (T+827353501 ms)
2025-02-17 09:45:09.289 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+827387109 ms)
2025-02-17 09:45:09.290 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+827387109 ms)
2025-02-17 09:45:09.290 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46516)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+827387109 ms)
2025-02-17 09:45:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827404418 ms)
2025-02-17 09:45:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827404419 ms)
2025-02-17 09:45:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:45:25.402762+00:00', 'runtime': '827400.003', 'adc_vb_f': '12.189', 'adc_va_f': '1.392', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827404422 ms)
2025-02-17 09:45:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:44:25.402853+00:00', 'runtime': '827340.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.6', 'ahl_t': '1588.6', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827404422 ms)
2025-02-17 09:45:35.748 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827413567 ms)
2025-02-17 09:45:35.749 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827413568 ms)
2025-02-17 09:45:35.753 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:45:25.400128+00:00'} (T+827413572 ms)
2025-02-17 09:46:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827464419 ms)
2025-02-17 09:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827464419 ms)
2025-02-17 09:46:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:46:25.400057+00:00', 'runtime': '827460.001', 'adc_vb_f': '12.183', 'adc_va_f': '1.401', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827464423 ms)
2025-02-17 09:46:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:45:25.402762+00:00', 'runtime': '827400.003', 'adc_vb_f': '12.189', 'adc_va_f': '1.392', 'adc_vl_f': '12.167', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.192', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827464424 ms)
2025-02-17 09:46:35.733 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827473553 ms)
2025-02-17 09:46:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827473553 ms)
2025-02-17 09:46:35.735 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:46:25.409123+00:00'} (T+827473554 ms)
2025-02-17 09:47:13.520 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+827511339 ms)
2025-02-17 09:47:13.520 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+827511339 ms)
2025-02-17 09:47:13.520 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46660)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+827511340 ms)
2025-02-17 09:47:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827524423 ms)
2025-02-17 09:47:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827524423 ms)
2025-02-17 09:47:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:47:25.400071+00:00', 'runtime': '827520.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.413', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827524426 ms)
2025-02-17 09:47:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:46:25.400057+00:00', 'runtime': '827460.001', 'adc_vb_f': '12.183', 'adc_va_f': '1.401', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827524427 ms)
2025-02-17 09:48:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827584407 ms)
2025-02-17 09:48:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827584407 ms)
2025-02-17 09:48:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:48:25.400675+00:00', 'runtime': '827580.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.413', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827584409 ms)
2025-02-17 09:48:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:47:25.400071+00:00', 'runtime': '827520.001', 'adc_vb_f': '12.195', 'adc_va_f': '1.413', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827584410 ms)
2025-02-17 09:48:27.687 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+827585506 ms)
2025-02-17 09:48:27.687 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+827585506 ms)
2025-02-17 09:48:27.688 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46740)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+827585507 ms)
2025-02-17 09:49:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827644407 ms)
2025-02-17 09:49:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827644408 ms)
2025-02-17 09:49:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:49:25.400164+00:00', 'runtime': '827640.001', 'adc_vb_f': '12.192', 'adc_va_f': '1.419', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827644412 ms)
2025-02-17 09:49:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:48:25.400675+00:00', 'runtime': '827580.001', 'adc_vb_f': '12.198', 'adc_va_f': '1.413', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827644413 ms)
2025-02-17 09:49:35.891 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827653710 ms)
2025-02-17 09:49:35.892 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827653711 ms)
2025-02-17 09:49:35.893 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:49:25.413048+00:00'} (T+827653712 ms)
2025-02-17 09:49:58.856 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+827676675 ms)
2025-02-17 09:49:58.856 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+827676675 ms)
2025-02-17 09:49:58.856 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46838)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+827676676 ms)
2025-02-17 09:50:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827704409 ms)
2025-02-17 09:50:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827704409 ms)
2025-02-17 09:50:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:50:25.402822+00:00', 'runtime': '827700.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.422', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827704413 ms)
2025-02-17 09:50:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:49:25.400164+00:00', 'runtime': '827640.001', 'adc_vb_f': '12.192', 'adc_va_f': '1.419', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827704413 ms)
2025-02-17 09:50:35.666 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827713485 ms)
2025-02-17 09:50:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827713486 ms)
2025-02-17 09:50:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:50:25.400226+00:00'} (T+827713487 ms)
2025-02-17 09:51:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827764411 ms)
2025-02-17 09:51:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827764411 ms)
2025-02-17 09:51:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:51:25.402821+00:00', 'runtime': '827760.003', 'adc_vb_f': '12.18', 'adc_va_f': '1.447', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827764416 ms)
2025-02-17 09:51:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:50:25.402822+00:00', 'runtime': '827700.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.422', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827764416 ms)
2025-02-17 09:51:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827773437 ms)
2025-02-17 09:51:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827773437 ms)
2025-02-17 09:51:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:51:25.400213+00:00'} (T+827773438 ms)
2025-02-17 09:52:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827824409 ms)
2025-02-17 09:52:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827824410 ms)
2025-02-17 09:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:52:25.401126+00:00', 'runtime': '827820.002', 'adc_vb_f': '12.183', 'adc_va_f': '1.456', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827824412 ms)
2025-02-17 09:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:51:25.402821+00:00', 'runtime': '827760.003', 'adc_vb_f': '12.18', 'adc_va_f': '1.447', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.189', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827824412 ms)
2025-02-17 09:52:35.721 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827833540 ms)
2025-02-17 09:52:35.721 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827833541 ms)
2025-02-17 09:52:35.722 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:52:25.400196+00:00'} (T+827833542 ms)
2025-02-17 09:53:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827884413 ms)
2025-02-17 09:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827884413 ms)
2025-02-17 09:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:53:25.400067+00:00', 'runtime': '827880.001', 'adc_vb_f': '12.183', 'adc_va_f': '1.471', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827884415 ms)
2025-02-17 09:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:52:25.401126+00:00', 'runtime': '827820.002', 'adc_vb_f': '12.183', 'adc_va_f': '1.456', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827884415 ms)
2025-02-17 09:53:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827893439 ms)
2025-02-17 09:53:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827893439 ms)
2025-02-17 09:53:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:53:25.409119+00:00'} (T+827893440 ms)
2025-02-17 09:54:06.217 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+827924037 ms)
2025-02-17 09:54:06.218 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+827924037 ms)
2025-02-17 09:54:06.218 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47102)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+827924037 ms)
2025-02-17 09:54:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+827944414 ms)
2025-02-17 09:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+827944414 ms)
2025-02-17 09:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:54:25.402795+00:00', 'runtime': '827940.003', 'adc_vb_f': '12.183', 'adc_va_f': '1.477', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827944416 ms)
2025-02-17 09:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:53:25.400067+00:00', 'runtime': '827880.001', 'adc_vb_f': '12.183', 'adc_va_f': '1.471', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20116', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+827944416 ms)
2025-02-17 09:54:35.716 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827953536 ms)
2025-02-17 09:54:35.717 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+827953536 ms)
2025-02-17 09:54:35.718 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:54:25.400177+00:00'} (T+827953537 ms)
2025-02-17 09:55:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828004417 ms)
2025-02-17 09:55:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828004417 ms)
2025-02-17 09:55:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:55:25.402683+00:00', 'runtime': '828000.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.471', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828004419 ms)
2025-02-17 09:55:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:54:25.402795+00:00', 'runtime': '827940.003', 'adc_vb_f': '12.183', 'adc_va_f': '1.477', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828004420 ms)
2025-02-17 09:55:35.799 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828013619 ms)
2025-02-17 09:55:35.800 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828013619 ms)
2025-02-17 09:55:35.801 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:55:25.400079+00:00'} (T+828013620 ms)
2025-02-17 09:55:40.417 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828018237 ms)
2025-02-17 09:55:40.418 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828018237 ms)
2025-02-17 09:55:40.418 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47204)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828018237 ms)
2025-02-17 09:55:50.513 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828028332 ms)
2025-02-17 09:55:50.513 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828028333 ms)
2025-02-17 09:55:50.514 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47228)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828028333 ms)
2025-02-17 09:56:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828064416 ms)
2025-02-17 09:56:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828064417 ms)
2025-02-17 09:56:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:56:25.402579+00:00', 'runtime': '828060.003', 'adc_vb_f': '12.192', 'adc_va_f': '1.459', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828064420 ms)
2025-02-17 09:56:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:55:25.402683+00:00', 'runtime': '828000.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.471', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.186', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828064421 ms)
2025-02-17 09:56:35.727 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828073547 ms)
2025-02-17 09:56:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828073547 ms)
2025-02-17 09:56:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:56:25.399989+00:00'} (T+828073552 ms)
2025-02-17 09:57:14.621 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828112440 ms)
2025-02-17 09:57:14.621 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828112440 ms)
2025-02-17 09:57:14.621 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828112441 ms)
2025-02-17 09:57:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828124418 ms)
2025-02-17 09:57:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828124418 ms)
2025-02-17 09:57:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:57:25.402675+00:00', 'runtime': '828120.003', 'adc_vb_f': '12.192', 'adc_va_f': '1.422', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828124421 ms)
2025-02-17 09:57:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:56:25.402579+00:00', 'runtime': '828060.003', 'adc_vb_f': '12.192', 'adc_va_f': '1.459', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828124422 ms)
2025-02-17 09:57:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828133435 ms)
2025-02-17 09:57:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828133436 ms)
2025-02-17 09:57:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:57:25.400062+00:00'} (T+828133437 ms)
2025-02-17 09:58:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828184418 ms)
2025-02-17 09:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828184419 ms)
2025-02-17 09:58:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:58:25.402795+00:00', 'runtime': '828180.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.395', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828184423 ms)
2025-02-17 09:58:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:57:25.402675+00:00', 'runtime': '828120.003', 'adc_vb_f': '12.192', 'adc_va_f': '1.422', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.7', 'ahl_t': '1588.7', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828184423 ms)
2025-02-17 09:58:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828193426 ms)
2025-02-17 09:58:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828193427 ms)
2025-02-17 09:58:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:58:25.400190+00:00'} (T+828193428 ms)
2025-02-17 09:59:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828244420 ms)
2025-02-17 09:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828244420 ms)
2025-02-17 09:59:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 09:59:25.400152+00:00', 'runtime': '828240.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828244424 ms)
2025-02-17 09:59:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:58:25.402795+00:00', 'runtime': '828180.003', 'adc_vb_f': '12.186', 'adc_va_f': '1.395', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828244425 ms)
2025-02-17 09:59:35.624 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828253443 ms)
2025-02-17 09:59:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828253444 ms)
2025-02-17 09:59:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 09:59:25.409118+00:00'} (T+828253445 ms)
2025-02-17 09:59:58.894 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828276714 ms)
2025-02-17 09:59:58.895 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828276714 ms)
2025-02-17 09:59:58.895 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47494)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828276714 ms)
2025-02-17 10:00:16.004 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828293823 ms)
2025-02-17 10:00:16.004 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828293823 ms)
2025-02-17 10:00:16.004 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47512)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828293824 ms)
2025-02-17 10:00:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828304422 ms)
2025-02-17 10:00:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828304422 ms)
2025-02-17 10:00:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:00:25.402879+00:00', 'runtime': '828300.003', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828304426 ms)
2025-02-17 10:00:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 09:59:25.400152+00:00', 'runtime': '828240.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828304426 ms)
2025-02-17 10:00:35.624 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828313444 ms)
2025-02-17 10:00:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828313444 ms)
2025-02-17 10:00:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:00:25.400254+00:00'} (T+828313445 ms)
2025-02-17 10:01:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828364423 ms)
2025-02-17 10:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828364423 ms)
2025-02-17 10:01:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:01:25.403162+00:00', 'runtime': '828360.004', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828364427 ms)
2025-02-17 10:01:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:00:25.402879+00:00', 'runtime': '828300.003', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828364428 ms)
2025-02-17 10:01:35.661 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828373481 ms)
2025-02-17 10:01:35.662 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828373481 ms)
2025-02-17 10:01:35.663 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:01:25.400523+00:00'} (T+828373482 ms)
2025-02-17 10:02:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828424422 ms)
2025-02-17 10:02:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828424422 ms)
2025-02-17 10:02:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:02:25.400196+00:00', 'runtime': '828420.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828424426 ms)
2025-02-17 10:02:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:01:25.403162+00:00', 'runtime': '828360.004', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828424427 ms)
2025-02-17 10:02:35.739 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828433558 ms)
2025-02-17 10:02:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828433559 ms)
2025-02-17 10:02:35.740 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:02:25.413128+00:00'} (T+828433560 ms)
2025-02-17 10:03:19.327 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828477147 ms)
2025-02-17 10:03:19.328 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828477147 ms)
2025-02-17 10:03:19.328 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47708)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828477147 ms)
2025-02-17 10:03:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828484422 ms)
2025-02-17 10:03:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828484423 ms)
2025-02-17 10:03:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:03:25.402663+00:00', 'runtime': '828480.003', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828484426 ms)
2025-02-17 10:03:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:02:25.400196+00:00', 'runtime': '828420.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828484427 ms)
2025-02-17 10:03:32.416 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828490236 ms)
2025-02-17 10:03:32.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828490236 ms)
2025-02-17 10:03:32.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47722)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828490236 ms)
2025-02-17 10:03:35.757 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828493576 ms)
2025-02-17 10:03:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828493577 ms)
2025-02-17 10:03:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:03:25.400079+00:00'} (T+828493578 ms)
2025-02-17 10:04:25.544 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828543363 ms)
2025-02-17 10:04:25.544 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828543364 ms)
2025-02-17 10:04:25.545 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47780)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828543364 ms)
2025-02-17 10:04:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828544422 ms)
2025-02-17 10:04:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828544423 ms)
2025-02-17 10:04:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:04:25.402677+00:00', 'runtime': '828540.003', 'adc_vb_f': '12.183', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828544424 ms)
2025-02-17 10:04:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:03:25.402663+00:00', 'runtime': '828480.003', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828544425 ms)
2025-02-17 10:04:35.644 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828553464 ms)
2025-02-17 10:04:35.645 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828553464 ms)
2025-02-17 10:04:35.646 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:04:25.400051+00:00'} (T+828553465 ms)
2025-02-17 10:05:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828604419 ms)
2025-02-17 10:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828604420 ms)
2025-02-17 10:05:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:05:25.402783+00:00', 'runtime': '828600.003', 'adc_vb_f': '12.183', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828604424 ms)
2025-02-17 10:05:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:04:25.402677+00:00', 'runtime': '828540.003', 'adc_vb_f': '12.183', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828604424 ms)
2025-02-17 10:05:35.855 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828613674 ms)
2025-02-17 10:05:35.855 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828613674 ms)
2025-02-17 10:05:35.856 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:05:25.400150+00:00'} (T+828613675 ms)
2025-02-17 10:05:39.640 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617459 ms)
2025-02-17 10:05:39.640 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617459 ms)
2025-02-17 10:05:39.641 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617460 ms)
2025-02-17 10:05:39.641 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617460 ms)
2025-02-17 10:05:39.641 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617461 ms)
2025-02-17 10:05:39.642 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617461 ms)
2025-02-17 10:05:39.642 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617461 ms)
2025-02-17 10:05:39.642 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617462 ms)
2025-02-17 10:05:39.643 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617462 ms)
2025-02-17 10:05:39.643 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617462 ms)
2025-02-17 10:05:39.643 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617462 ms)
2025-02-17 10:05:39.644 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617463 ms)
2025-02-17 10:05:39.644 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617463 ms)
2025-02-17 10:05:39.644 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617463 ms)
2025-02-17 10:05:39.644 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617464 ms)
2025-02-17 10:05:39.645 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617464 ms)
2025-02-17 10:05:39.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617464 ms)
2025-02-17 10:05:39.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617465 ms)
2025-02-17 10:05:39.646 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617465 ms)
2025-02-17 10:05:39.646 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617465 ms)
2025-02-17 10:05:39.646 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617466 ms)
2025-02-17 10:05:39.647 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617466 ms)
2025-02-17 10:05:39.647 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617466 ms)
2025-02-17 10:05:39.647 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617466 ms)
2025-02-17 10:05:39.648 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617467 ms)
2025-02-17 10:05:39.648 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617467 ms)
2025-02-17 10:05:39.648 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617467 ms)
2025-02-17 10:05:39.648 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+828617468 ms)
2025-02-17 10:05:39.649 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+828617468 ms)
2025-02-17 10:05:39.650 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828617469 ms)
2025-02-17 10:06:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828664419 ms)
2025-02-17 10:06:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828664419 ms)
2025-02-17 10:06:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:06:25.400152+00:00', 'runtime': '828660.001', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828664423 ms)
2025-02-17 10:06:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:05:25.402783+00:00', 'runtime': '828600.003', 'adc_vb_f': '12.183', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828664423 ms)
2025-02-17 10:06:36.014 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828673834 ms)
2025-02-17 10:06:36.015 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828673834 ms)
2025-02-17 10:06:36.016 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:06:25.413127+00:00'} (T+828673835 ms)
2025-02-17 10:07:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828724420 ms)
2025-02-17 10:07:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828724420 ms)
2025-02-17 10:07:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:07:25.400170+00:00', 'runtime': '828720.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828724421 ms)
2025-02-17 10:07:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:06:25.400152+00:00', 'runtime': '828660.001', 'adc_vb_f': '12.186', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828724422 ms)
2025-02-17 10:07:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828733439 ms)
2025-02-17 10:07:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828733440 ms)
2025-02-17 10:07:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:07:25.400298+00:00'} (T+828733441 ms)
2025-02-17 10:08:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828784419 ms)
2025-02-17 10:08:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828784420 ms)
2025-02-17 10:08:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:08:25.402827+00:00', 'runtime': '828780.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828784424 ms)
2025-02-17 10:08:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:07:25.400170+00:00', 'runtime': '828720.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828784424 ms)
2025-02-17 10:08:35.726 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828793545 ms)
2025-02-17 10:08:35.727 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828793546 ms)
2025-02-17 10:08:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:08:25.400207+00:00'} (T+828793547 ms)
2025-02-17 10:08:41.955 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+828799774 ms)
2025-02-17 10:08:41.955 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+828799775 ms)
2025-02-17 10:08:41.956 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48066)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+828799775 ms)
2025-02-17 10:09:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828844405 ms)
2025-02-17 10:09:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828844406 ms)
2025-02-17 10:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:09:25.402700+00:00', 'runtime': '828840.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828844410 ms)
2025-02-17 10:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:08:25.402827+00:00', 'runtime': '828780.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828844410 ms)
2025-02-17 10:09:36.491 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828854311 ms)
2025-02-17 10:09:36.492 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828854311 ms)
2025-02-17 10:09:36.493 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:09:25.400066+00:00'} (T+828854313 ms)
2025-02-17 10:10:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828904408 ms)
2025-02-17 10:10:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828904408 ms)
2025-02-17 10:10:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:10:25.402826+00:00', 'runtime': '828900.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828904412 ms)
2025-02-17 10:10:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:09:25.402700+00:00', 'runtime': '828840.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828904413 ms)
2025-02-17 10:10:35.693 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828913512 ms)
2025-02-17 10:10:35.693 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828913513 ms)
2025-02-17 10:10:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:10:25.400211+00:00'} (T+828913513 ms)
2025-02-17 10:11:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+828964406 ms)
2025-02-17 10:11:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+828964406 ms)
2025-02-17 10:11:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:11:25.400180+00:00', 'runtime': '828960.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.346', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828964408 ms)
2025-02-17 10:11:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:10:25.402826+00:00', 'runtime': '828900.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.8', 'ahl_t': '1588.8', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+828964409 ms)
2025-02-17 10:11:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828973429 ms)
2025-02-17 10:11:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+828973429 ms)
2025-02-17 10:11:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:11:25.400099+00:00'} (T+828973430 ms)
2025-02-17 10:12:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829024408 ms)
2025-02-17 10:12:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829024409 ms)
2025-02-17 10:12:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:12:25.400183+00:00', 'runtime': '829020.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.294', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829024410 ms)
2025-02-17 10:12:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:11:25.400180+00:00', 'runtime': '828960.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.346', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829024410 ms)
2025-02-17 10:12:34.244 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032063 ms)
2025-02-17 10:12:34.244 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032064 ms)
2025-02-17 10:12:34.245 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032064 ms)
2025-02-17 10:12:34.245 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032065 ms)
2025-02-17 10:12:34.245 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032065 ms)
2025-02-17 10:12:34.246 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032065 ms)
2025-02-17 10:12:34.246 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032066 ms)
2025-02-17 10:12:34.246 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032066 ms)
2025-02-17 10:12:34.247 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032066 ms)
2025-02-17 10:12:34.247 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032067 ms)
2025-02-17 10:12:34.247 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032067 ms)
2025-02-17 10:12:34.248 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032067 ms)
2025-02-17 10:12:34.248 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032068 ms)
2025-02-17 10:12:34.249 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032068 ms)
2025-02-17 10:12:34.249 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032068 ms)
2025-02-17 10:12:34.250 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032069 ms)
2025-02-17 10:12:34.250 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032069 ms)
2025-02-17 10:12:34.250 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032069 ms)
2025-02-17 10:12:34.250 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032070 ms)
2025-02-17 10:12:34.251 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032070 ms)
2025-02-17 10:12:34.251 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032070 ms)
2025-02-17 10:12:34.251 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032071 ms)
2025-02-17 10:12:34.252 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032071 ms)
2025-02-17 10:12:34.252 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032071 ms)
2025-02-17 10:12:34.252 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+829032072 ms)
2025-02-17 10:12:34.252 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+829032072 ms)
2025-02-17 10:12:34.253 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829032072 ms)
2025-02-17 10:12:35.785 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829033604 ms)
2025-02-17 10:12:35.785 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829033605 ms)
2025-02-17 10:12:35.787 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:12:25.400208+00:00'} (T+829033606 ms)
2025-02-17 10:12:52.356 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829050175 ms)
2025-02-17 10:12:52.356 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829050176 ms)
2025-02-17 10:12:52.357 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48336)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829050176 ms)
2025-02-17 10:13:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829084407 ms)
2025-02-17 10:13:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829084407 ms)
2025-02-17 10:13:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:13:25.400196+00:00', 'runtime': '829080.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.285', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829084408 ms)
2025-02-17 10:13:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:12:25.400183+00:00', 'runtime': '829020.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.294', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829084409 ms)
2025-02-17 10:13:35.741 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829093560 ms)
2025-02-17 10:13:35.741 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829093561 ms)
2025-02-17 10:13:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:13:25.400002+00:00'} (T+829093562 ms)
2025-02-17 10:14:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829144424 ms)
2025-02-17 10:14:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829144425 ms)
2025-02-17 10:14:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:14:25.400214+00:00', 'runtime': '829140.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.285', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829144426 ms)
2025-02-17 10:14:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:13:25.400196+00:00', 'runtime': '829080.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.285', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829144426 ms)
2025-02-17 10:14:35.697 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829153517 ms)
2025-02-17 10:14:35.698 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829153517 ms)
2025-02-17 10:14:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:14:25.400406+00:00'} (T+829153518 ms)
2025-02-17 10:15:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829204412 ms)
2025-02-17 10:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829204412 ms)
2025-02-17 10:15:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:15:25.400143+00:00', 'runtime': '829200.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.34', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829204413 ms)
2025-02-17 10:15:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:14:25.400214+00:00', 'runtime': '829140.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.285', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829204414 ms)
2025-02-17 10:15:35.429 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829213248 ms)
2025-02-17 10:15:35.430 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829213249 ms)
2025-02-17 10:15:35.431 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:15:25.400098+00:00'} (T+829213250 ms)
2025-02-17 10:16:32.242 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829270061 ms)
2025-02-17 10:16:32.242 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829270061 ms)
2025-02-17 10:16:32.243 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:16:25.400156+00:00', 'runtime': '829260.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.346', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829270062 ms)
2025-02-17 10:16:32.244 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:15:25.400143+00:00', 'runtime': '829200.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.34', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829270063 ms)
2025-02-17 10:17:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829324415 ms)
2025-02-17 10:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829324415 ms)
2025-02-17 10:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:17:25.400069+00:00', 'runtime': '829320.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829324416 ms)
2025-02-17 10:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:16:25.400156+00:00', 'runtime': '829260.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.346', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829324417 ms)
2025-02-17 10:18:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829384416 ms)
2025-02-17 10:18:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829384416 ms)
2025-02-17 10:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:18:25.400186+00:00', 'runtime': '829380.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829384417 ms)
2025-02-17 10:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:17:25.400069+00:00', 'runtime': '829320.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829384418 ms)
2025-02-17 10:19:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829444419 ms)
2025-02-17 10:19:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829444419 ms)
2025-02-17 10:19:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:19:25.400168+00:00', 'runtime': '829440.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829444420 ms)
2025-02-17 10:19:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:18:25.400186+00:00', 'runtime': '829380.001', 'adc_vb_f': '12.18', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829444421 ms)
2025-02-17 10:19:35.697 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829453516 ms)
2025-02-17 10:19:35.698 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829453517 ms)
2025-02-17 10:19:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:19:25.409042+00:00'} (T+829453518 ms)
2025-02-17 10:20:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829504421 ms)
2025-02-17 10:20:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829504421 ms)
2025-02-17 10:20:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:20:25.402689+00:00', 'runtime': '829500.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829504422 ms)
2025-02-17 10:20:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:19:25.400168+00:00', 'runtime': '829440.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829504423 ms)
2025-02-17 10:20:35.642 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829513462 ms)
2025-02-17 10:20:35.643 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829513462 ms)
2025-02-17 10:20:35.647 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:20:25.400087+00:00'} (T+829513467 ms)
2025-02-17 10:21:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829564420 ms)
2025-02-17 10:21:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829564421 ms)
2025-02-17 10:21:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:21:25.400162+00:00', 'runtime': '829560.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829564425 ms)
2025-02-17 10:21:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:20:25.402689+00:00', 'runtime': '829500.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.18', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829564425 ms)
2025-02-17 10:21:34.056 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829571875 ms)
2025-02-17 10:21:34.056 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829571876 ms)
2025-02-17 10:21:34.057 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48896)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829571876 ms)
2025-02-17 10:21:35.644 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829573463 ms)
2025-02-17 10:21:35.645 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829573464 ms)
2025-02-17 10:21:35.649 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:21:25.400144+00:00'} (T+829573468 ms)
2025-02-17 10:22:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829624421 ms)
2025-02-17 10:22:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829624421 ms)
2025-02-17 10:22:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:22:25.400157+00:00', 'runtime': '829620.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829624425 ms)
2025-02-17 10:22:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:21:25.400162+00:00', 'runtime': '829560.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829624426 ms)
2025-02-17 10:22:35.653 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829633473 ms)
2025-02-17 10:22:35.654 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829633473 ms)
2025-02-17 10:22:35.655 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:22:25.400142+00:00'} (T+829633475 ms)
2025-02-17 10:23:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829684405 ms)
2025-02-17 10:23:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829684405 ms)
2025-02-17 10:23:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:23:25.400177+00:00', 'runtime': '829680.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829684410 ms)
2025-02-17 10:23:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:22:25.400157+00:00', 'runtime': '829620.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1588.9', 'ahl_t': '1588.9', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829684410 ms)
2025-02-17 10:23:48.275 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829706094 ms)
2025-02-17 10:23:48.275 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829706094 ms)
2025-02-17 10:23:48.276 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49042)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829706095 ms)
2025-02-17 10:24:20.384 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829738203 ms)
2025-02-17 10:24:20.384 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829738203 ms)
2025-02-17 10:24:20.385 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49078)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829738204 ms)
2025-02-17 10:24:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829744407 ms)
2025-02-17 10:24:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829744407 ms)
2025-02-17 10:24:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:24:25.400183+00:00', 'runtime': '829740.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829744408 ms)
2025-02-17 10:24:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:23:25.400177+00:00', 'runtime': '829680.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829744409 ms)
2025-02-17 10:24:34.475 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829752294 ms)
2025-02-17 10:24:34.475 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829752294 ms)
2025-02-17 10:24:34.475 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49094)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829752295 ms)
2025-02-17 10:25:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829804406 ms)
2025-02-17 10:25:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829804407 ms)
2025-02-17 10:25:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:25:25.400207+00:00', 'runtime': '829800.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829804408 ms)
2025-02-17 10:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:24:25.400183+00:00', 'runtime': '829740.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829804408 ms)
2025-02-17 10:25:35.898 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829813717 ms)
2025-02-17 10:25:35.899 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829813718 ms)
2025-02-17 10:25:35.900 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:25:25.400345+00:00'} (T+829813720 ms)
2025-02-17 10:26:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829864406 ms)
2025-02-17 10:26:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829864407 ms)
2025-02-17 10:26:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:26:25.400161+00:00', 'runtime': '829860.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829864408 ms)
2025-02-17 10:26:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:25:25.400207+00:00', 'runtime': '829800.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829864408 ms)
2025-02-17 10:26:36.025 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829873844 ms)
2025-02-17 10:26:36.025 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829873845 ms)
2025-02-17 10:26:36.026 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:26:25.400396+00:00'} (T+829873846 ms)
2025-02-17 10:26:55.742 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829893561 ms)
2025-02-17 10:26:55.742 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829893562 ms)
2025-02-17 10:26:55.743 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49256)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829893562 ms)
2025-02-17 10:27:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829924409 ms)
2025-02-17 10:27:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829924409 ms)
2025-02-17 10:27:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:27:25.400172+00:00', 'runtime': '829920.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829924410 ms)
2025-02-17 10:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:26:25.400161+00:00', 'runtime': '829860.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829924411 ms)
2025-02-17 10:27:35.675 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829933494 ms)
2025-02-17 10:27:35.675 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829933495 ms)
2025-02-17 10:27:35.676 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:27:25.413123+00:00'} (T+829933496 ms)
2025-02-17 10:27:57.882 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829955701 ms)
2025-02-17 10:27:57.882 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829955701 ms)
2025-02-17 10:27:57.883 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49324)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829955702 ms)
2025-02-17 10:28:13.987 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+829971806 ms)
2025-02-17 10:28:13.987 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+829971806 ms)
2025-02-17 10:28:13.987 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49342)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+829971806 ms)
2025-02-17 10:28:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+829984408 ms)
2025-02-17 10:28:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+829984409 ms)
2025-02-17 10:28:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:28:25.400203+00:00', 'runtime': '829980.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829984410 ms)
2025-02-17 10:28:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:27:25.400172+00:00', 'runtime': '829920.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+829984411 ms)
2025-02-17 10:28:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829993426 ms)
2025-02-17 10:28:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+829993426 ms)
2025-02-17 10:28:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:28:25.400122+00:00'} (T+829993427 ms)
2025-02-17 10:29:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830044412 ms)
2025-02-17 10:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830044412 ms)
2025-02-17 10:29:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:29:25.400221+00:00', 'runtime': '830040.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830044413 ms)
2025-02-17 10:29:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:28:25.400203+00:00', 'runtime': '829980.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830044414 ms)
2025-02-17 10:29:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830053424 ms)
2025-02-17 10:29:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830053425 ms)
2025-02-17 10:29:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:29:25.400231+00:00'} (T+830053426 ms)
2025-02-17 10:30:16.214 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830094034 ms)
2025-02-17 10:30:16.215 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830094034 ms)
2025-02-17 10:30:16.215 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49474)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830094034 ms)
2025-02-17 10:30:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830104409 ms)
2025-02-17 10:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830104410 ms)
2025-02-17 10:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:30:25.400197+00:00', 'runtime': '830100.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830104411 ms)
2025-02-17 10:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:29:25.400221+00:00', 'runtime': '830040.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830104411 ms)
2025-02-17 10:30:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830113428 ms)
2025-02-17 10:30:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830113429 ms)
2025-02-17 10:30:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:30:25.400223+00:00'} (T+830113430 ms)
2025-02-17 10:31:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830164410 ms)
2025-02-17 10:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830164411 ms)
2025-02-17 10:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:31:25.400188+00:00', 'runtime': '830160.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830164412 ms)
2025-02-17 10:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:30:25.400197+00:00', 'runtime': '830100.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830164413 ms)
2025-02-17 10:31:56.404 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830194223 ms)
2025-02-17 10:31:56.404 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830194224 ms)
2025-02-17 10:31:56.405 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49584)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830194224 ms)
2025-02-17 10:32:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830224410 ms)
2025-02-17 10:32:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830224410 ms)
2025-02-17 10:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:32:25.400163+00:00', 'runtime': '830220.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830224411 ms)
2025-02-17 10:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:31:25.400188+00:00', 'runtime': '830160.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830224412 ms)
2025-02-17 10:33:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830284412 ms)
2025-02-17 10:33:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830284413 ms)
2025-02-17 10:33:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:33:25.400163+00:00', 'runtime': '830280.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830284414 ms)
2025-02-17 10:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:32:25.400163+00:00', 'runtime': '830220.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830284414 ms)
2025-02-17 10:33:35.712 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830293532 ms)
2025-02-17 10:33:35.713 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830293532 ms)
2025-02-17 10:33:35.714 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:33:25.400330+00:00'} (T+830293533 ms)
2025-02-17 10:34:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830344413 ms)
2025-02-17 10:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830344413 ms)
2025-02-17 10:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:34:25.400173+00:00', 'runtime': '830340.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830344414 ms)
2025-02-17 10:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:33:25.400163+00:00', 'runtime': '830280.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830344415 ms)
2025-02-17 10:35:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830404412 ms)
2025-02-17 10:35:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830404413 ms)
2025-02-17 10:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:35:25.400174+00:00', 'runtime': '830400.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830404414 ms)
2025-02-17 10:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:34:25.400173+00:00', 'runtime': '830340.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830404414 ms)
2025-02-17 10:35:31.774 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830409593 ms)
2025-02-17 10:35:31.774 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830409593 ms)
2025-02-17 10:35:31.775 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49810)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830409594 ms)
2025-02-17 10:36:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830464415 ms)
2025-02-17 10:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830464416 ms)
2025-02-17 10:36:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:36:25.400181+00:00', 'runtime': '830460.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830464417 ms)
2025-02-17 10:36:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:35:25.400174+00:00', 'runtime': '830400.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.0', 'ahl_t': '1589.0', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830464418 ms)
2025-02-17 10:36:38.950 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830476770 ms)
2025-02-17 10:36:38.951 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830476770 ms)
2025-02-17 10:36:38.951 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49892)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830476771 ms)
2025-02-17 10:37:10.073 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830507892 ms)
2025-02-17 10:37:10.073 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830507893 ms)
2025-02-17 10:37:10.074 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49926)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830507893 ms)
2025-02-17 10:37:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830524413 ms)
2025-02-17 10:37:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830524413 ms)
2025-02-17 10:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:37:25.400151+00:00', 'runtime': '830520.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830524415 ms)
2025-02-17 10:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:36:25.400181+00:00', 'runtime': '830460.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830524415 ms)
2025-02-17 10:37:32.172 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830529992 ms)
2025-02-17 10:37:32.173 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830529992 ms)
2025-02-17 10:37:32.173 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49950)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830529992 ms)
2025-02-17 10:37:39.235 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830537055 ms)
2025-02-17 10:37:39.236 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830537055 ms)
2025-02-17 10:37:39.236 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49958)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830537055 ms)
2025-02-17 10:37:53.315 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830551134 ms)
2025-02-17 10:37:53.315 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830551135 ms)
2025-02-17 10:37:53.316 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49976)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830551135 ms)
2025-02-17 10:38:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830584414 ms)
2025-02-17 10:38:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830584415 ms)
2025-02-17 10:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:38:25.400164+00:00', 'runtime': '830580.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830584416 ms)
2025-02-17 10:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:37:25.400151+00:00', 'runtime': '830520.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830584416 ms)
2025-02-17 10:38:27.423 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830585242 ms)
2025-02-17 10:38:27.423 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830585242 ms)
2025-02-17 10:38:27.424 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50012)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830585243 ms)
2025-02-17 10:39:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830644416 ms)
2025-02-17 10:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830644416 ms)
2025-02-17 10:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:39:25.400191+00:00', 'runtime': '830640.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830644417 ms)
2025-02-17 10:39:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:38:25.400164+00:00', 'runtime': '830580.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830644418 ms)
2025-02-17 10:39:32.575 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830650394 ms)
2025-02-17 10:39:32.575 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830650394 ms)
2025-02-17 10:39:32.575 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50080)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830650394 ms)
2025-02-17 10:39:34.625 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830652444 ms)
2025-02-17 10:39:34.625 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830652445 ms)
2025-02-17 10:39:34.626 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50086)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830652445 ms)
2025-02-17 10:39:34.665 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830652484 ms)
2025-02-17 10:39:34.665 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830652485 ms)
2025-02-17 10:39:34.670 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50088)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830652489 ms)
2025-02-17 10:39:35.853 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830653672 ms)
2025-02-17 10:39:35.854 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830653673 ms)
2025-02-17 10:39:35.854 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:39:25.400338+00:00'} (T+830653674 ms)
2025-02-17 10:40:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830704419 ms)
2025-02-17 10:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830704419 ms)
2025-02-17 10:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:40:25.400171+00:00', 'runtime': '830700.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830704420 ms)
2025-02-17 10:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:39:25.400191+00:00', 'runtime': '830640.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830704421 ms)
2025-02-17 10:41:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830764406 ms)
2025-02-17 10:41:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830764407 ms)
2025-02-17 10:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:41:25.400198+00:00', 'runtime': '830760.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830764408 ms)
2025-02-17 10:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:40:25.400171+00:00', 'runtime': '830700.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830764408 ms)
2025-02-17 10:42:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830824410 ms)
2025-02-17 10:42:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830824410 ms)
2025-02-17 10:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:42:25.400323+00:00', 'runtime': '830820.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830824411 ms)
2025-02-17 10:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:41:25.400198+00:00', 'runtime': '830760.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830824412 ms)
2025-02-17 10:43:09.020 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830866840 ms)
2025-02-17 10:43:09.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830866840 ms)
2025-02-17 10:43:09.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50314)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830866840 ms)
2025-02-17 10:43:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830884411 ms)
2025-02-17 10:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830884411 ms)
2025-02-17 10:43:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:43:25.400168+00:00', 'runtime': '830880.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830884412 ms)
2025-02-17 10:43:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:42:25.400323+00:00', 'runtime': '830820.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830884413 ms)
2025-02-17 10:43:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830893430 ms)
2025-02-17 10:43:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830893431 ms)
2025-02-17 10:43:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:43:25.400341+00:00'} (T+830893431 ms)
2025-02-17 10:43:38.151 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830895970 ms)
2025-02-17 10:43:38.151 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830895970 ms)
2025-02-17 10:43:38.151 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50346)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830895971 ms)
2025-02-17 10:44:17.289 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830935108 ms)
2025-02-17 10:44:17.289 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830935109 ms)
2025-02-17 10:44:17.290 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50388)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830935109 ms)
2025-02-17 10:44:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+830944412 ms)
2025-02-17 10:44:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+830944412 ms)
2025-02-17 10:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:44:25.400184+00:00', 'runtime': '830940.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830944413 ms)
2025-02-17 10:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:43:25.400168+00:00', 'runtime': '830880.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+830944413 ms)
2025-02-17 10:44:35.741 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830953560 ms)
2025-02-17 10:44:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+830953561 ms)
2025-02-17 10:44:35.743 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:44:25.400506+00:00'} (T+830953562 ms)
2025-02-17 10:44:55.402 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+830973221 ms)
2025-02-17 10:44:55.402 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+830973221 ms)
2025-02-17 10:44:55.402 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50432)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+830973222 ms)
2025-02-17 10:45:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831004415 ms)
2025-02-17 10:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831004415 ms)
2025-02-17 10:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:45:25.400176+00:00', 'runtime': '831000.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831004416 ms)
2025-02-17 10:45:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:44:25.400184+00:00', 'runtime': '830940.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831004417 ms)
2025-02-17 10:45:45.516 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+831023335 ms)
2025-02-17 10:45:45.516 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+831023335 ms)
2025-02-17 10:45:45.516 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50492)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+831023335 ms)
2025-02-17 10:46:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831064403 ms)
2025-02-17 10:46:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831064403 ms)
2025-02-17 10:46:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:46:25.400204+00:00', 'runtime': '831060.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831064404 ms)
2025-02-17 10:46:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:45:25.400176+00:00', 'runtime': '831000.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831064405 ms)
2025-02-17 10:47:09.696 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+831107516 ms)
2025-02-17 10:47:09.697 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+831107516 ms)
2025-02-17 10:47:09.697 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50586)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+831107516 ms)
2025-02-17 10:47:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831124419 ms)
2025-02-17 10:47:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831124420 ms)
2025-02-17 10:47:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:47:25.400177+00:00', 'runtime': '831120.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831124421 ms)
2025-02-17 10:47:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:46:25.400204+00:00', 'runtime': '831060.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831124421 ms)
2025-02-17 10:47:35.897 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831133716 ms)
2025-02-17 10:47:35.898 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831133717 ms)
2025-02-17 10:47:35.898 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:47:25.400680+00:00'} (T+831133718 ms)
2025-02-17 10:47:48.807 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+831146627 ms)
2025-02-17 10:47:48.808 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+831146627 ms)
2025-02-17 10:47:48.808 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50628)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+831146628 ms)
2025-02-17 10:48:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831184406 ms)
2025-02-17 10:48:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831184406 ms)
2025-02-17 10:48:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:48:25.400154+00:00', 'runtime': '831180.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831184407 ms)
2025-02-17 10:48:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:47:25.400177+00:00', 'runtime': '831120.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831184408 ms)
2025-02-17 10:49:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831244406 ms)
2025-02-17 10:49:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831244406 ms)
2025-02-17 10:49:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:49:25.399992+00:00', 'runtime': '831240.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831244411 ms)
2025-02-17 10:49:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:48:25.400154+00:00', 'runtime': '831180.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.1', 'ahl_t': '1589.1', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831244411 ms)
2025-02-17 10:50:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831304422 ms)
2025-02-17 10:50:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831304423 ms)
2025-02-17 10:50:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:50:25.400207+00:00', 'runtime': '831300.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831304427 ms)
2025-02-17 10:50:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:49:25.399992+00:00', 'runtime': '831240.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831304427 ms)
2025-02-17 10:51:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831364409 ms)
2025-02-17 10:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831364409 ms)
2025-02-17 10:51:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:51:25.400068+00:00', 'runtime': '831360.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831364413 ms)
2025-02-17 10:51:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:50:25.400207+00:00', 'runtime': '831300.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831364414 ms)
2025-02-17 10:52:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831424409 ms)
2025-02-17 10:52:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831424409 ms)
2025-02-17 10:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:52:25.400176+00:00', 'runtime': '831420.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831424414 ms)
2025-02-17 10:52:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:51:25.400068+00:00', 'runtime': '831360.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831424414 ms)
2025-02-17 10:53:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831484411 ms)
2025-02-17 10:53:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831484411 ms)
2025-02-17 10:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:53:25.400185+00:00', 'runtime': '831480.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831484415 ms)
2025-02-17 10:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:52:25.400176+00:00', 'runtime': '831420.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831484415 ms)
2025-02-17 10:54:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831544414 ms)
2025-02-17 10:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831544415 ms)
2025-02-17 10:54:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:54:25.400196+00:00', 'runtime': '831540.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831544418 ms)
2025-02-17 10:54:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:53:25.400185+00:00', 'runtime': '831480.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20117', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831544419 ms)
2025-02-17 10:55:10.435 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+831588254 ms)
2025-02-17 10:55:10.435 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+831588254 ms)
2025-02-17 10:55:10.435 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51092)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+831588255 ms)
2025-02-17 10:55:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831604413 ms)
2025-02-17 10:55:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831604414 ms)
2025-02-17 10:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:55:25.400171+00:00', 'runtime': '831600.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831604416 ms)
2025-02-17 10:55:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:54:25.400196+00:00', 'runtime': '831540.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831604416 ms)
2025-02-17 10:55:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831613433 ms)
2025-02-17 10:55:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831613434 ms)
2025-02-17 10:55:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:55:25.400346+00:00'} (T+831613439 ms)
2025-02-17 10:55:37.573 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+831615393 ms)
2025-02-17 10:55:37.574 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+831615393 ms)
2025-02-17 10:55:37.574 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51122)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+831615393 ms)
2025-02-17 10:56:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831664414 ms)
2025-02-17 10:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831664414 ms)
2025-02-17 10:56:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:56:25.400179+00:00', 'runtime': '831660.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831664416 ms)
2025-02-17 10:56:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:55:25.400171+00:00', 'runtime': '831600.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831664417 ms)
2025-02-17 10:56:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831673439 ms)
2025-02-17 10:56:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831673440 ms)
2025-02-17 10:56:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 10:56:25.400194+00:00'} (T+831673446 ms)
2025-02-17 10:57:07.753 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+831705572 ms)
2025-02-17 10:57:07.753 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+831705572 ms)
2025-02-17 10:57:07.753 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51230)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+831705573 ms)
2025-02-17 10:57:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831724415 ms)
2025-02-17 10:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831724416 ms)
2025-02-17 10:57:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:57:25.400176+00:00', 'runtime': '831720.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831724418 ms)
2025-02-17 10:57:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:56:25.400179+00:00', 'runtime': '831660.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831724418 ms)
2025-02-17 10:57:48.873 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+831746692 ms)
2025-02-17 10:57:48.873 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+831746693 ms)
2025-02-17 10:57:48.874 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51274)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+831746693 ms)
2025-02-17 10:58:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831784402 ms)
2025-02-17 10:58:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831784403 ms)
2025-02-17 10:58:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:58:25.400604+00:00', 'runtime': '831780.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831784404 ms)
2025-02-17 10:58:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:57:25.400176+00:00', 'runtime': '831720.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831784404 ms)
2025-02-17 10:59:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831844404 ms)
2025-02-17 10:59:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831844404 ms)
2025-02-17 10:59:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 10:59:25.400632+00:00', 'runtime': '831840.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831844405 ms)
2025-02-17 10:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:58:25.400604+00:00', 'runtime': '831780.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831844406 ms)
2025-02-17 11:00:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831904421 ms)
2025-02-17 11:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831904421 ms)
2025-02-17 11:00:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:00:25.402829+00:00', 'runtime': '831900.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831904422 ms)
2025-02-17 11:00:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 10:59:25.400632+00:00', 'runtime': '831840.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831904423 ms)
2025-02-17 11:00:35.456 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831913275 ms)
2025-02-17 11:00:35.456 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831913276 ms)
2025-02-17 11:00:35.459 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:00:25.400207+00:00'} (T+831913278 ms)
2025-02-17 11:01:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+831964420 ms)
2025-02-17 11:01:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+831964420 ms)
2025-02-17 11:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:01:25.402808+00:00', 'runtime': '831960.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831964421 ms)
2025-02-17 11:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:00:25.402829+00:00', 'runtime': '831900.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+831964422 ms)
2025-02-17 11:01:35.457 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831973277 ms)
2025-02-17 11:01:35.458 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+831973277 ms)
2025-02-17 11:01:35.460 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:01:25.400198+00:00'} (T+831973279 ms)
2025-02-17 11:02:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832024421 ms)
2025-02-17 11:02:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832024422 ms)
2025-02-17 11:02:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:02:25.402862+00:00', 'runtime': '832020.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832024423 ms)
2025-02-17 11:02:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:01:25.402808+00:00', 'runtime': '831960.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.2', 'ahl_t': '1589.2', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832024424 ms)
2025-02-17 11:03:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832084405 ms)
2025-02-17 11:03:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832084406 ms)
2025-02-17 11:03:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:03:25.400202+00:00', 'runtime': '832080.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832084407 ms)
2025-02-17 11:03:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:02:25.402862+00:00', 'runtime': '832020.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832084407 ms)
2025-02-17 11:03:53.404 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832111223 ms)
2025-02-17 11:03:53.404 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832111224 ms)
2025-02-17 11:03:53.405 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51660)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832111224 ms)
2025-02-17 11:04:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832144405 ms)
2025-02-17 11:04:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832144406 ms)
2025-02-17 11:04:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:04:25.400179+00:00', 'runtime': '832140.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832144407 ms)
2025-02-17 11:04:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:03:25.400202+00:00', 'runtime': '832080.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832144407 ms)
2025-02-17 11:04:35.724 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832153543 ms)
2025-02-17 11:04:35.724 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832153544 ms)
2025-02-17 11:04:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:04:25.400330+00:00'} (T+832153548 ms)
2025-02-17 11:04:40.516 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832158336 ms)
2025-02-17 11:04:40.517 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832158336 ms)
2025-02-17 11:04:40.517 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51710)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832158336 ms)
2025-02-17 11:04:46.586 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832164405 ms)
2025-02-17 11:04:46.586 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832164405 ms)
2025-02-17 11:04:46.586 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51718)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832164406 ms)
2025-02-17 11:05:04.676 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832182496 ms)
2025-02-17 11:05:04.677 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832182496 ms)
2025-02-17 11:05:04.677 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51740)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832182496 ms)
2025-02-17 11:05:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832204407 ms)
2025-02-17 11:05:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832204407 ms)
2025-02-17 11:05:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:05:25.400184+00:00', 'runtime': '832200.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832204408 ms)
2025-02-17 11:05:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:04:25.400179+00:00', 'runtime': '832140.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832204409 ms)
2025-02-17 11:05:35.894 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832213714 ms)
2025-02-17 11:05:35.895 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832213714 ms)
2025-02-17 11:05:35.899 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:05:25.400063+00:00'} (T+832213719 ms)
2025-02-17 11:06:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832264407 ms)
2025-02-17 11:06:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832264407 ms)
2025-02-17 11:06:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:06:25.400170+00:00', 'runtime': '832260.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832264411 ms)
2025-02-17 11:06:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:05:25.400184+00:00', 'runtime': '832200.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832264412 ms)
2025-02-17 11:06:35.665 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832273484 ms)
2025-02-17 11:06:35.665 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832273485 ms)
2025-02-17 11:06:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:06:25.400407+00:00'} (T+832273486 ms)
2025-02-17 11:07:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832324409 ms)
2025-02-17 11:07:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832324409 ms)
2025-02-17 11:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:07:25.402720+00:00', 'runtime': '832320.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832324414 ms)
2025-02-17 11:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:06:25.400170+00:00', 'runtime': '832260.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832324415 ms)
2025-02-17 11:07:35.655 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832333474 ms)
2025-02-17 11:07:35.655 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832333475 ms)
2025-02-17 11:07:35.657 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:07:25.400135+00:00'} (T+832333476 ms)
2025-02-17 11:08:25.008 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832382827 ms)
2025-02-17 11:08:25.008 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832382827 ms)
2025-02-17 11:08:25.008 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51964)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832382828 ms)
2025-02-17 11:08:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832384408 ms)
2025-02-17 11:08:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832384408 ms)
2025-02-17 11:08:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:08:25.400239+00:00', 'runtime': '832380.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832384409 ms)
2025-02-17 11:08:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:07:25.402720+00:00', 'runtime': '832320.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832384410 ms)
2025-02-17 11:08:35.677 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832393496 ms)
2025-02-17 11:08:35.677 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832393496 ms)
2025-02-17 11:08:35.679 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:08:25.400235+00:00'} (T+832393498 ms)
2025-02-17 11:09:25.158 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832442977 ms)
2025-02-17 11:09:25.158 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832442977 ms)
2025-02-17 11:09:25.158 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52030)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832442977 ms)
2025-02-17 11:09:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832444410 ms)
2025-02-17 11:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832444410 ms)
2025-02-17 11:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:09:25.400181+00:00', 'runtime': '832440.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.379', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832444411 ms)
2025-02-17 11:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:08:25.400239+00:00', 'runtime': '832380.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832444412 ms)
2025-02-17 11:09:35.830 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832453650 ms)
2025-02-17 11:09:35.831 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832453650 ms)
2025-02-17 11:09:35.832 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:09:25.400117+00:00'} (T+832453651 ms)
2025-02-17 11:10:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832504415 ms)
2025-02-17 11:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832504415 ms)
2025-02-17 11:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:10:25.400177+00:00', 'runtime': '832500.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.385', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832504416 ms)
2025-02-17 11:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:09:25.400181+00:00', 'runtime': '832440.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.379', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832504417 ms)
2025-02-17 11:10:35.731 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832513550 ms)
2025-02-17 11:10:35.731 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832513551 ms)
2025-02-17 11:10:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:10:25.399994+00:00'} (T+832513552 ms)
2025-02-17 11:11:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832564414 ms)
2025-02-17 11:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832564414 ms)
2025-02-17 11:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:11:25.400175+00:00', 'runtime': '832560.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832564415 ms)
2025-02-17 11:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:10:25.400177+00:00', 'runtime': '832500.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.385', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832564416 ms)
2025-02-17 11:11:47.405 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832585224 ms)
2025-02-17 11:11:47.405 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832585225 ms)
2025-02-17 11:11:47.406 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52184)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832585225 ms)
2025-02-17 11:12:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832624415 ms)
2025-02-17 11:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832624415 ms)
2025-02-17 11:12:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:12:25.400165+00:00', 'runtime': '832620.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832624416 ms)
2025-02-17 11:12:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:11:25.400175+00:00', 'runtime': '832560.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832624417 ms)
2025-02-17 11:13:02.597 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832660417 ms)
2025-02-17 11:13:02.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832660417 ms)
2025-02-17 11:13:02.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52264)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832660417 ms)
2025-02-17 11:13:24.624 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+832682444 ms)
2025-02-17 11:13:24.625 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+832682444 ms)
2025-02-17 11:13:24.625 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832682444 ms)
2025-02-17 11:13:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832684417 ms)
2025-02-17 11:13:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832684417 ms)
2025-02-17 11:13:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:13:25.400159+00:00', 'runtime': '832680.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832684418 ms)
2025-02-17 11:13:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:12:25.400165+00:00', 'runtime': '832620.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832684419 ms)
2025-02-17 11:14:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832744415 ms)
2025-02-17 11:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832744415 ms)
2025-02-17 11:14:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:14:25.400257+00:00', 'runtime': '832740.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832744417 ms)
2025-02-17 11:14:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:13:25.400159+00:00', 'runtime': '832680.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832744417 ms)
2025-02-17 11:15:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832804416 ms)
2025-02-17 11:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832804417 ms)
2025-02-17 11:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:15:25.400172+00:00', 'runtime': '832800.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832804418 ms)
2025-02-17 11:15:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:14:25.400257+00:00', 'runtime': '832740.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.3', 'ahl_t': '1589.3', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832804418 ms)
2025-02-17 11:16:22.926 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832860745 ms)
2025-02-17 11:16:22.926 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832860745 ms)
2025-02-17 11:16:22.927 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52484)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832860746 ms)
2025-02-17 11:16:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832864416 ms)
2025-02-17 11:16:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832864416 ms)
2025-02-17 11:16:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:16:25.400172+00:00', 'runtime': '832860.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.349', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832864417 ms)
2025-02-17 11:16:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:15:25.400172+00:00', 'runtime': '832800.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832864419 ms)
2025-02-17 11:16:36.473 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832874292 ms)
2025-02-17 11:16:36.473 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832874293 ms)
2025-02-17 11:16:36.474 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:16:25.400343+00:00'} (T+832874294 ms)
2025-02-17 11:17:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832924419 ms)
2025-02-17 11:17:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832924419 ms)
2025-02-17 11:17:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:17:25.400158+00:00', 'runtime': '832920.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.343', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832924420 ms)
2025-02-17 11:17:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:16:25.400172+00:00', 'runtime': '832860.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.349', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832924421 ms)
2025-02-17 11:17:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832933435 ms)
2025-02-17 11:17:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832933436 ms)
2025-02-17 11:17:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:17:25.400347+00:00'} (T+832933437 ms)
2025-02-17 11:18:26.177 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832983996 ms)
2025-02-17 11:18:26.177 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832983996 ms)
2025-02-17 11:18:26.177 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52616)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832983997 ms)
2025-02-17 11:18:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+832984417 ms)
2025-02-17 11:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+832984418 ms)
2025-02-17 11:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:18:25.400178+00:00', 'runtime': '832980.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.346', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832984419 ms)
2025-02-17 11:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:17:25.400158+00:00', 'runtime': '832920.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.343', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+832984419 ms)
2025-02-17 11:18:35.254 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+832993073 ms)
2025-02-17 11:18:35.254 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+832993074 ms)
2025-02-17 11:18:35.255 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52628)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+832993074 ms)
2025-02-17 11:18:35.675 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832993495 ms)
2025-02-17 11:18:35.676 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+832993495 ms)
2025-02-17 11:18:35.677 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:18:25.400198+00:00'} (T+832993496 ms)
2025-02-17 11:18:52.323 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833010142 ms)
2025-02-17 11:18:52.323 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833010143 ms)
2025-02-17 11:18:52.324 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52648)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833010143 ms)
2025-02-17 11:19:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833044416 ms)
2025-02-17 11:19:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833044417 ms)
2025-02-17 11:19:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:19:25.400160+00:00', 'runtime': '833040.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.343', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833044418 ms)
2025-02-17 11:19:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:18:25.400178+00:00', 'runtime': '832980.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.346', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833044418 ms)
2025-02-17 11:19:36.211 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833054030 ms)
2025-02-17 11:19:36.211 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833054031 ms)
2025-02-17 11:19:36.212 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:19:25.400126+00:00'} (T+833054032 ms)
2025-02-17 11:20:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833104417 ms)
2025-02-17 11:20:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833104418 ms)
2025-02-17 11:20:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:20:25.400169+00:00', 'runtime': '833100.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.315', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833104419 ms)
2025-02-17 11:20:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:19:25.400160+00:00', 'runtime': '833040.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.343', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833104419 ms)
2025-02-17 11:20:36.030 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833113849 ms)
2025-02-17 11:20:36.031 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833113850 ms)
2025-02-17 11:20:36.032 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:20:25.400087+00:00'} (T+833113851 ms)
2025-02-17 11:21:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833164418 ms)
2025-02-17 11:21:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833164419 ms)
2025-02-17 11:21:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:21:25.400160+00:00', 'runtime': '833160.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.312', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833164420 ms)
2025-02-17 11:21:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:20:25.400169+00:00', 'runtime': '833100.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.315', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833164420 ms)
2025-02-17 11:21:34.573 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833172392 ms)
2025-02-17 11:21:34.573 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833172393 ms)
2025-02-17 11:21:34.574 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52822)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833172393 ms)
2025-02-17 11:21:35.627 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833173447 ms)
2025-02-17 11:21:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833173447 ms)
2025-02-17 11:21:35.629 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:21:25.400301+00:00'} (T+833173448 ms)
2025-02-17 11:22:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833224417 ms)
2025-02-17 11:22:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833224418 ms)
2025-02-17 11:22:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:22:25.400162+00:00', 'runtime': '833220.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.303', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833224419 ms)
2025-02-17 11:22:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:21:25.400160+00:00', 'runtime': '833160.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.312', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833224419 ms)
2025-02-17 11:22:35.744 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833233564 ms)
2025-02-17 11:22:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833233564 ms)
2025-02-17 11:22:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:22:25.400178+00:00'} (T+833233565 ms)
2025-02-17 11:23:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833284416 ms)
2025-02-17 11:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833284417 ms)
2025-02-17 11:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:23:25.400179+00:00', 'runtime': '833280.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.306', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833284418 ms)
2025-02-17 11:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:22:25.400162+00:00', 'runtime': '833220.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.303', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833284418 ms)
2025-02-17 11:23:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833293423 ms)
2025-02-17 11:23:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833293424 ms)
2025-02-17 11:23:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:23:25.400058+00:00'} (T+833293425 ms)
2025-02-17 11:24:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833344413 ms)
2025-02-17 11:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833344414 ms)
2025-02-17 11:24:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:24:25.400218+00:00', 'runtime': '833340.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.318', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833344415 ms)
2025-02-17 11:24:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:23:25.400179+00:00', 'runtime': '833280.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.306', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833344415 ms)
2025-02-17 11:24:32.871 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833350690 ms)
2025-02-17 11:24:32.871 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833350690 ms)
2025-02-17 11:24:32.872 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53012)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833350691 ms)
2025-02-17 11:24:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833353435 ms)
2025-02-17 11:24:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833353436 ms)
2025-02-17 11:24:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:24:25.400230+00:00'} (T+833353437 ms)
2025-02-17 11:25:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833404415 ms)
2025-02-17 11:25:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833404415 ms)
2025-02-17 11:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:25:25.400181+00:00', 'runtime': '833400.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.312', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833404416 ms)
2025-02-17 11:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:24:25.400218+00:00', 'runtime': '833340.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.318', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833404417 ms)
2025-02-17 11:25:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833413434 ms)
2025-02-17 11:25:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833413434 ms)
2025-02-17 11:25:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:25:25.400544+00:00'} (T+833413435 ms)
2025-02-17 11:26:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833464416 ms)
2025-02-17 11:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833464417 ms)
2025-02-17 11:26:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:26:25.400176+00:00', 'runtime': '833460.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.343', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833464418 ms)
2025-02-17 11:26:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:25:25.400181+00:00', 'runtime': '833400.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.312', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833464418 ms)
2025-02-17 11:26:35.698 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833473518 ms)
2025-02-17 11:26:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833473518 ms)
2025-02-17 11:26:35.700 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:26:25.400184+00:00'} (T+833473519 ms)
2025-02-17 11:27:06.172 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833503991 ms)
2025-02-17 11:27:06.172 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833503992 ms)
2025-02-17 11:27:06.173 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53188)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833503992 ms)
2025-02-17 11:27:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833524415 ms)
2025-02-17 11:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833524415 ms)
2025-02-17 11:27:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:27:25.402686+00:00', 'runtime': '833520.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.349', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833524416 ms)
2025-02-17 11:27:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:26:25.400176+00:00', 'runtime': '833460.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.343', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.498', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833524417 ms)
2025-02-17 11:27:35.623 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833533442 ms)
2025-02-17 11:27:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833533442 ms)
2025-02-17 11:27:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:27:25.400095+00:00'} (T+833533445 ms)
2025-02-17 11:28:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833584412 ms)
2025-02-17 11:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833584413 ms)
2025-02-17 11:28:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:28:25.402797+00:00', 'runtime': '833580.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.364', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833584414 ms)
2025-02-17 11:28:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:27:25.402686+00:00', 'runtime': '833520.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.349', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.4', 'ahl_t': '1589.4', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833584414 ms)
2025-02-17 11:28:29.352 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833587172 ms)
2025-02-17 11:28:29.353 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833587172 ms)
2025-02-17 11:28:29.354 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53276)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833587173 ms)
2025-02-17 11:28:35.742 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833593562 ms)
2025-02-17 11:28:35.743 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833593562 ms)
2025-02-17 11:28:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:28:25.400187+00:00'} (T+833593565 ms)
2025-02-17 11:29:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833644414 ms)
2025-02-17 11:29:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833644414 ms)
2025-02-17 11:29:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:29:25.400068+00:00', 'runtime': '833640.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833644415 ms)
2025-02-17 11:29:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:28:25.402797+00:00', 'runtime': '833580.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.364', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833644415 ms)
2025-02-17 11:29:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833653431 ms)
2025-02-17 11:29:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833653432 ms)
2025-02-17 11:29:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:29:25.405128+00:00'} (T+833653433 ms)
2025-02-17 11:30:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833704413 ms)
2025-02-17 11:30:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833704414 ms)
2025-02-17 11:30:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:30:25.400173+00:00', 'runtime': '833700.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833704415 ms)
2025-02-17 11:30:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:29:25.400068+00:00', 'runtime': '833640.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833704415 ms)
2025-02-17 11:30:35.713 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833713532 ms)
2025-02-17 11:30:35.713 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833713533 ms)
2025-02-17 11:30:35.718 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:30:25.400179+00:00'} (T+833713538 ms)
2025-02-17 11:31:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833764415 ms)
2025-02-17 11:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833764415 ms)
2025-02-17 11:31:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:31:25.400152+00:00', 'runtime': '833760.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833764416 ms)
2025-02-17 11:31:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:30:25.400173+00:00', 'runtime': '833700.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833764417 ms)
2025-02-17 11:31:35.761 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833773580 ms)
2025-02-17 11:31:35.761 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833773581 ms)
2025-02-17 11:31:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:31:25.400078+00:00'} (T+833773582 ms)
2025-02-17 11:31:46.703 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833784522 ms)
2025-02-17 11:31:46.703 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833784522 ms)
2025-02-17 11:31:46.703 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53486)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833784523 ms)
2025-02-17 11:32:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833824412 ms)
2025-02-17 11:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833824413 ms)
2025-02-17 11:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:32:25.400167+00:00', 'runtime': '833820.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.379', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833824414 ms)
2025-02-17 11:32:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:31:25.400152+00:00', 'runtime': '833760.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833824415 ms)
2025-02-17 11:32:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833833422 ms)
2025-02-17 11:32:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833833423 ms)
2025-02-17 11:32:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:32:25.400024+00:00'} (T+833833428 ms)
2025-02-17 11:33:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833884415 ms)
2025-02-17 11:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833884416 ms)
2025-02-17 11:33:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:33:25.400192+00:00', 'runtime': '833880.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.376', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833884420 ms)
2025-02-17 11:33:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:32:25.400167+00:00', 'runtime': '833820.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.379', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833884420 ms)
2025-02-17 11:33:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833893434 ms)
2025-02-17 11:33:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833893434 ms)
2025-02-17 11:33:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:33:25.400217+00:00'} (T+833893439 ms)
2025-02-17 11:34:10.966 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833928785 ms)
2025-02-17 11:34:10.966 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833928786 ms)
2025-02-17 11:34:10.967 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53642)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833928786 ms)
2025-02-17 11:34:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+833944416 ms)
2025-02-17 11:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+833944416 ms)
2025-02-17 11:34:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:34:25.400183+00:00', 'runtime': '833940.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833944419 ms)
2025-02-17 11:34:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:33:25.400192+00:00', 'runtime': '833880.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.376', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+833944420 ms)
2025-02-17 11:34:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833953429 ms)
2025-02-17 11:34:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+833953430 ms)
2025-02-17 11:34:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:34:25.400184+00:00'} (T+833953434 ms)
2025-02-17 11:35:08.132 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+833985951 ms)
2025-02-17 11:35:08.132 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+833985952 ms)
2025-02-17 11:35:08.133 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53704)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+833985952 ms)
2025-02-17 11:35:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834004417 ms)
2025-02-17 11:35:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834004417 ms)
2025-02-17 11:35:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:35:25.400166+00:00', 'runtime': '834000.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834004421 ms)
2025-02-17 11:35:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:34:25.400183+00:00', 'runtime': '833940.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834004421 ms)
2025-02-17 11:35:35.689 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834013509 ms)
2025-02-17 11:35:35.690 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834013509 ms)
2025-02-17 11:35:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:35:25.400186+00:00'} (T+834013511 ms)
2025-02-17 11:36:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834064420 ms)
2025-02-17 11:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834064421 ms)
2025-02-17 11:36:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:36:25.400197+00:00', 'runtime': '834060.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834064422 ms)
2025-02-17 11:36:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:35:25.400166+00:00', 'runtime': '834000.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.389', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834064423 ms)
2025-02-17 11:36:35.749 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834073568 ms)
2025-02-17 11:36:35.750 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834073569 ms)
2025-02-17 11:36:35.751 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:36:25.400131+00:00'} (T+834073571 ms)
2025-02-17 11:37:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834124417 ms)
2025-02-17 11:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834124417 ms)
2025-02-17 11:37:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:37:25.400163+00:00', 'runtime': '834120.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834124420 ms)
2025-02-17 11:37:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:36:25.400197+00:00', 'runtime': '834060.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834124420 ms)
2025-02-17 11:37:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834133431 ms)
2025-02-17 11:37:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834133431 ms)
2025-02-17 11:37:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:37:25.400030+00:00'} (T+834133434 ms)
2025-02-17 11:37:52.411 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+834150231 ms)
2025-02-17 11:37:52.411 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+834150231 ms)
2025-02-17 11:37:52.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53892)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+834150231 ms)
2025-02-17 11:38:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834184419 ms)
2025-02-17 11:38:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834184420 ms)
2025-02-17 11:38:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:38:25.400163+00:00', 'runtime': '834180.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834184422 ms)
2025-02-17 11:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:37:25.400163+00:00', 'runtime': '834120.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834184422 ms)
2025-02-17 11:38:35.743 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834193562 ms)
2025-02-17 11:38:35.743 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834193563 ms)
2025-02-17 11:38:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:38:25.400187+00:00'} (T+834193564 ms)
2025-02-17 11:39:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834244417 ms)
2025-02-17 11:39:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834244418 ms)
2025-02-17 11:39:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:39:25.400193+00:00', 'runtime': '834240.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834244419 ms)
2025-02-17 11:39:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:38:25.400163+00:00', 'runtime': '834180.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.177', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834244419 ms)
2025-02-17 11:39:36.043 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834253863 ms)
2025-02-17 11:39:36.044 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834253863 ms)
2025-02-17 11:39:36.045 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:39:25.400067+00:00'} (T+834253864 ms)
2025-02-17 11:40:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834304403 ms)
2025-02-17 11:40:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834304404 ms)
2025-02-17 11:40:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:40:25.400185+00:00', 'runtime': '834300.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834304405 ms)
2025-02-17 11:40:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:39:25.400193+00:00', 'runtime': '834240.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834304405 ms)
2025-02-17 11:40:35.750 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834313569 ms)
2025-02-17 11:40:35.750 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834313570 ms)
2025-02-17 11:40:35.755 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:40:25.400274+00:00'} (T+834313574 ms)
2025-02-17 11:41:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834364419 ms)
2025-02-17 11:41:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834364419 ms)
2025-02-17 11:41:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:41:25.400204+00:00', 'runtime': '834360.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834364420 ms)
2025-02-17 11:41:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:40:25.400185+00:00', 'runtime': '834300.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.5', 'ahl_t': '1589.5', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834364421 ms)
2025-02-17 11:41:29.744 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+834367564 ms)
2025-02-17 11:41:29.745 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+834367564 ms)
2025-02-17 11:41:29.745 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54122)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+834367564 ms)
2025-02-17 11:41:35.888 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834373708 ms)
2025-02-17 11:41:35.889 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834373708 ms)
2025-02-17 11:41:35.893 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:41:25.400125+00:00'} (T+834373713 ms)
2025-02-17 11:42:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834424418 ms)
2025-02-17 11:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834424418 ms)
2025-02-17 11:42:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:42:25.400220+00:00', 'runtime': '834420.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.413', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834424419 ms)
2025-02-17 11:42:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:41:25.400204+00:00', 'runtime': '834360.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.392', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834424420 ms)
2025-02-17 11:42:26.912 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+834424731 ms)
2025-02-17 11:42:26.912 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+834424732 ms)
2025-02-17 11:42:26.913 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54184)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+834424732 ms)
2025-02-17 11:42:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834433430 ms)
2025-02-17 11:42:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834433431 ms)
2025-02-17 11:42:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:42:25.400444+00:00'} (T+834433436 ms)
2025-02-17 11:43:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834484404 ms)
2025-02-17 11:43:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834484404 ms)
2025-02-17 11:43:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:43:25.400180+00:00', 'runtime': '834480.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.44', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834484406 ms)
2025-02-17 11:43:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:42:25.400220+00:00', 'runtime': '834420.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.413', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834484406 ms)
2025-02-17 11:43:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834493430 ms)
2025-02-17 11:43:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834493431 ms)
2025-02-17 11:43:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:43:25.400183+00:00'} (T+834493436 ms)
2025-02-17 11:44:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834544403 ms)
2025-02-17 11:44:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834544403 ms)
2025-02-17 11:44:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:44:25.400196+00:00', 'runtime': '834540.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.48', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834544405 ms)
2025-02-17 11:44:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:43:25.400180+00:00', 'runtime': '834480.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.44', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834544406 ms)
2025-02-17 11:44:35.727 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834553547 ms)
2025-02-17 11:44:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834553547 ms)
2025-02-17 11:44:35.733 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:44:25.400184+00:00'} (T+834553552 ms)
2025-02-17 11:45:16.760 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+834594580 ms)
2025-02-17 11:45:16.761 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+834594580 ms)
2025-02-17 11:45:16.761 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54366)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+834594580 ms)
2025-02-17 11:45:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834604418 ms)
2025-02-17 11:45:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834604419 ms)
2025-02-17 11:45:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:45:25.402721+00:00', 'runtime': '834600.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.498', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834604421 ms)
2025-02-17 11:45:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:44:25.400196+00:00', 'runtime': '834540.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.48', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834604421 ms)
2025-02-17 11:45:55.893 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+834633712 ms)
2025-02-17 11:45:55.893 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+834633713 ms)
2025-02-17 11:45:55.894 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54418)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+834633713 ms)
2025-02-17 11:46:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834664404 ms)
2025-02-17 11:46:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834664405 ms)
2025-02-17 11:46:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:46:25.400831+00:00', 'runtime': '834660.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.501', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834664406 ms)
2025-02-17 11:46:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:45:25.402721+00:00', 'runtime': '834600.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.498', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.505', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.173', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834664407 ms)
2025-02-17 11:47:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834724419 ms)
2025-02-17 11:47:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834724420 ms)
2025-02-17 11:47:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:47:25.402767+00:00', 'runtime': '834720.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.505', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834724422 ms)
2025-02-17 11:47:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:46:25.400831+00:00', 'runtime': '834660.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.501', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834724422 ms)
2025-02-17 11:47:35.826 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834733645 ms)
2025-02-17 11:47:35.827 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834733646 ms)
2025-02-17 11:47:35.828 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:47:25.400139+00:00'} (T+834733647 ms)
2025-02-17 11:48:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834784418 ms)
2025-02-17 11:48:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834784419 ms)
2025-02-17 11:48:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:48:25.402639+00:00', 'runtime': '834780.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.566', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834784420 ms)
2025-02-17 11:48:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:47:25.402767+00:00', 'runtime': '834720.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.505', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834784420 ms)
2025-02-17 11:49:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834844402 ms)
2025-02-17 11:49:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834844402 ms)
2025-02-17 11:49:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:49:25.400182+00:00', 'runtime': '834840.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.608', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834844407 ms)
2025-02-17 11:49:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:48:25.402639+00:00', 'runtime': '834780.003', 'adc_vb_f': '12.177', 'adc_va_f': '1.566', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834844407 ms)
2025-02-17 11:49:40.251 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+834858070 ms)
2025-02-17 11:49:40.251 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+834858070 ms)
2025-02-17 11:49:40.252 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54654)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+834858072 ms)
2025-02-17 11:50:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834904419 ms)
2025-02-17 11:50:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834904420 ms)
2025-02-17 11:50:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:50:25.400184+00:00', 'runtime': '834900.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.608', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834904421 ms)
2025-02-17 11:50:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:49:25.400182+00:00', 'runtime': '834840.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.608', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834904421 ms)
2025-02-17 11:50:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834913437 ms)
2025-02-17 11:50:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+834913438 ms)
2025-02-17 11:50:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:50:25.400664+00:00'} (T+834913439 ms)
2025-02-17 11:51:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+834964403 ms)
2025-02-17 11:51:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+834964403 ms)
2025-02-17 11:51:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:51:25.400164+00:00', 'runtime': '834960.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.715', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834964404 ms)
2025-02-17 11:51:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:50:25.400184+00:00', 'runtime': '834900.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.608', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+834964405 ms)
2025-02-17 11:52:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835024405 ms)
2025-02-17 11:52:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835024405 ms)
2025-02-17 11:52:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:52:25.400164+00:00', 'runtime': '835020.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.755', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835024406 ms)
2025-02-17 11:52:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:51:25.400164+00:00', 'runtime': '834960.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.715', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835024407 ms)
2025-02-17 11:52:31.561 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835029380 ms)
2025-02-17 11:52:31.561 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835029381 ms)
2025-02-17 11:52:31.562 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54836)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835029381 ms)
2025-02-17 11:53:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835084404 ms)
2025-02-17 11:53:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835084405 ms)
2025-02-17 11:53:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:53:25.400175+00:00', 'runtime': '835080.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.822', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835084406 ms)
2025-02-17 11:53:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:52:25.400164+00:00', 'runtime': '835020.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.755', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.6', 'ahl_t': '1589.6', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835084406 ms)
2025-02-17 11:53:32.736 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835090556 ms)
2025-02-17 11:53:32.737 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835090556 ms)
2025-02-17 11:53:32.737 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54900)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835090556 ms)
2025-02-17 11:53:35.684 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835093504 ms)
2025-02-17 11:53:35.685 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835093504 ms)
2025-02-17 11:53:35.686 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:53:25.400342+00:00'} (T+835093505 ms)
2025-02-17 11:54:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835144406 ms)
2025-02-17 11:54:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835144406 ms)
2025-02-17 11:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:54:25.400209+00:00', 'runtime': '835140.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.929', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835144407 ms)
2025-02-17 11:54:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:53:25.400175+00:00', 'runtime': '835080.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.822', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20118', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835144409 ms)
2025-02-17 11:54:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835153425 ms)
2025-02-17 11:54:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835153425 ms)
2025-02-17 11:54:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:54:25.400125+00:00'} (T+835153426 ms)
2025-02-17 11:55:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835204422 ms)
2025-02-17 11:55:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835204422 ms)
2025-02-17 11:55:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:55:25.402850+00:00', 'runtime': '835200.003', 'adc_vb_f': '12.177', 'adc_va_f': '2.097', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835204424 ms)
2025-02-17 11:55:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:54:25.400209+00:00', 'runtime': '835140.001', 'adc_vb_f': '12.177', 'adc_va_f': '1.929', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835204425 ms)
2025-02-17 11:55:52.986 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835230805 ms)
2025-02-17 11:55:52.986 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835230805 ms)
2025-02-17 11:55:52.986 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55064)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835230806 ms)
2025-02-17 11:56:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835264407 ms)
2025-02-17 11:56:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835264407 ms)
2025-02-17 11:56:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:56:25.400784+00:00', 'runtime': '835260.001', 'adc_vb_f': '12.177', 'adc_va_f': '2.252', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835264410 ms)
2025-02-17 11:56:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:55:25.402850+00:00', 'runtime': '835200.003', 'adc_vb_f': '12.177', 'adc_va_f': '2.097', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.17', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835264410 ms)
2025-02-17 11:57:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835324407 ms)
2025-02-17 11:57:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835324408 ms)
2025-02-17 11:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:57:25.400601+00:00', 'runtime': '835320.001', 'adc_vb_f': '12.177', 'adc_va_f': '2.466', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835324412 ms)
2025-02-17 11:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:56:25.400784+00:00', 'runtime': '835260.001', 'adc_vb_f': '12.177', 'adc_va_f': '2.252', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.488', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835324412 ms)
2025-02-17 11:58:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835384421 ms)
2025-02-17 11:58:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835384422 ms)
2025-02-17 11:58:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:58:25.400188+00:00', 'runtime': '835380.001', 'adc_vb_f': '12.177', 'adc_va_f': '2.74', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835384424 ms)
2025-02-17 11:58:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:57:25.400601+00:00', 'runtime': '835320.001', 'adc_vb_f': '12.177', 'adc_va_f': '2.466', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835384424 ms)
2025-02-17 11:58:35.673 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835393493 ms)
2025-02-17 11:58:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835393493 ms)
2025-02-17 11:58:35.675 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:58:25.413047+00:00'} (T+835393494 ms)
2025-02-17 11:59:03.222 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835421041 ms)
2025-02-17 11:59:03.222 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835421041 ms)
2025-02-17 11:59:03.222 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835421041 ms)
2025-02-17 11:59:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835444419 ms)
2025-02-17 11:59:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835444420 ms)
2025-02-17 11:59:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 11:59:25.402805+00:00', 'runtime': '835440.003', 'adc_vb_f': '12.177', 'adc_va_f': '3.003', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835444421 ms)
2025-02-17 11:59:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:58:25.400188+00:00', 'runtime': '835380.001', 'adc_vb_f': '12.177', 'adc_va_f': '2.74', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835444422 ms)
2025-02-17 11:59:35.685 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835453504 ms)
2025-02-17 11:59:35.685 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835453505 ms)
2025-02-17 11:59:35.686 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 11:59:25.400181+00:00'} (T+835453506 ms)
2025-02-17 11:59:38.375 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835456195 ms)
2025-02-17 11:59:38.376 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835456195 ms)
2025-02-17 11:59:38.376 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55300)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835456195 ms)
2025-02-17 12:00:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835504424 ms)
2025-02-17 12:00:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835504424 ms)
2025-02-17 12:00:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:00:25.400093+00:00', 'runtime': '835500.001', 'adc_vb_f': '12.177', 'adc_va_f': '3.394', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835504428 ms)
2025-02-17 12:00:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 11:59:25.402805+00:00', 'runtime': '835440.003', 'adc_vb_f': '12.177', 'adc_va_f': '3.003', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835504429 ms)
2025-02-17 12:00:35.912 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835513731 ms)
2025-02-17 12:00:35.913 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835513732 ms)
2025-02-17 12:00:35.914 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:00:25.413120+00:00'} (T+835513733 ms)
2025-02-17 12:00:39.554 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835517373 ms)
2025-02-17 12:00:39.554 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835517374 ms)
2025-02-17 12:00:39.555 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55366)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835517374 ms)
2025-02-17 12:01:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835564421 ms)
2025-02-17 12:01:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835564422 ms)
2025-02-17 12:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:01:25.402782+00:00', 'runtime': '835560.003', 'adc_vb_f': '12.177', 'adc_va_f': '3.763', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835564423 ms)
2025-02-17 12:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:00:25.400093+00:00', 'runtime': '835500.001', 'adc_vb_f': '12.177', 'adc_va_f': '3.394', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835564423 ms)
2025-02-17 12:01:35.743 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835573562 ms)
2025-02-17 12:01:35.743 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835573563 ms)
2025-02-17 12:01:35.744 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:01:25.400200+00:00'} (T+835573564 ms)
2025-02-17 12:01:53.649 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835591469 ms)
2025-02-17 12:01:53.650 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835591469 ms)
2025-02-17 12:01:53.650 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835591470 ms)
2025-02-17 12:01:53.651 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835591471 ms)
2025-02-17 12:01:53.652 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835591471 ms)
2025-02-17 12:01:53.652 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835591471 ms)
2025-02-17 12:01:53.652 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835591472 ms)
2025-02-17 12:01:53.653 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835591472 ms)
2025-02-17 12:01:53.653 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835591472 ms)
2025-02-17 12:02:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835624420 ms)
2025-02-17 12:02:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835624421 ms)
2025-02-17 12:02:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:02:25.400535+00:00', 'runtime': '835620.001', 'adc_vb_f': '12.177', 'adc_va_f': '4.205', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835624425 ms)
2025-02-17 12:02:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:01:25.402782+00:00', 'runtime': '835560.003', 'adc_vb_f': '12.177', 'adc_va_f': '3.763', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.167', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835624425 ms)
2025-02-17 12:02:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835633438 ms)
2025-02-17 12:02:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835633439 ms)
2025-02-17 12:02:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:02:25.400049+00:00'} (T+835633444 ms)
2025-02-17 12:03:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835684419 ms)
2025-02-17 12:03:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835684419 ms)
2025-02-17 12:03:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:03:25.400532+00:00', 'runtime': '835680.001', 'adc_vb_f': '12.177', 'adc_va_f': '4.718', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835684420 ms)
2025-02-17 12:03:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:02:25.400535+00:00', 'runtime': '835620.001', 'adc_vb_f': '12.177', 'adc_va_f': '4.205', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835684420 ms)
2025-02-17 12:04:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835744417 ms)
2025-02-17 12:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835744418 ms)
2025-02-17 12:04:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:04:25.400169+00:00', 'runtime': '835740.001', 'adc_vb_f': '12.177', 'adc_va_f': '5.362', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835744419 ms)
2025-02-17 12:04:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:03:25.400532+00:00', 'runtime': '835680.001', 'adc_vb_f': '12.177', 'adc_va_f': '4.718', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835744419 ms)
2025-02-17 12:05:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835804416 ms)
2025-02-17 12:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835804416 ms)
2025-02-17 12:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:05:25.400193+00:00', 'runtime': '835800.001', 'adc_vb_f': '12.177', 'adc_va_f': '6.07', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835804417 ms)
2025-02-17 12:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:04:25.400169+00:00', 'runtime': '835740.001', 'adc_vb_f': '12.177', 'adc_va_f': '5.362', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835804418 ms)
2025-02-17 12:06:22.056 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835859875 ms)
2025-02-17 12:06:22.056 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835859875 ms)
2025-02-17 12:06:22.056 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55740)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835859876 ms)
2025-02-17 12:06:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835864415 ms)
2025-02-17 12:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835864416 ms)
2025-02-17 12:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:06:25.400157+00:00', 'runtime': '835860.001', 'adc_vb_f': '12.177', 'adc_va_f': '6.772', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835864417 ms)
2025-02-17 12:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:05:25.400193+00:00', 'runtime': '835800.001', 'adc_vb_f': '12.177', 'adc_va_f': '6.07', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.5', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.7', 'ahl_t': '1589.7', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835864417 ms)
2025-02-17 12:06:45.184 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835883003 ms)
2025-02-17 12:06:45.184 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835883003 ms)
2025-02-17 12:06:45.184 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55764)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835883004 ms)
2025-02-17 12:07:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835924413 ms)
2025-02-17 12:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835924413 ms)
2025-02-17 12:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:07:25.400175+00:00', 'runtime': '835920.001', 'adc_vb_f': '12.177', 'adc_va_f': '7.596', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835924414 ms)
2025-02-17 12:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:06:25.400157+00:00', 'runtime': '835860.001', 'adc_vb_f': '12.177', 'adc_va_f': '6.772', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835924415 ms)
2025-02-17 12:07:29.240 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835927059 ms)
2025-02-17 12:07:29.240 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835927060 ms)
2025-02-17 12:07:29.241 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835927060 ms)
2025-02-17 12:07:29.242 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835927061 ms)
2025-02-17 12:07:29.242 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835927061 ms)
2025-02-17 12:07:29.243 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835927062 ms)
2025-02-17 12:07:29.244 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835927063 ms)
2025-02-17 12:07:29.245 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835927064 ms)
2025-02-17 12:07:29.246 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835927065 ms)
2025-02-17 12:07:29.248 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835927067 ms)
2025-02-17 12:07:29.248 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835927067 ms)
2025-02-17 12:07:29.249 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835927068 ms)
2025-02-17 12:07:29.251 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+835927070 ms)
2025-02-17 12:07:29.252 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+835927071 ms)
2025-02-17 12:07:29.253 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835927072 ms)
2025-02-17 12:07:35.741 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835933560 ms)
2025-02-17 12:07:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+835933561 ms)
2025-02-17 12:07:35.750 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:07:25.400347+00:00'} (T+835933569 ms)
2025-02-17 12:08:13.376 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+835971195 ms)
2025-02-17 12:08:13.376 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+835971195 ms)
2025-02-17 12:08:13.377 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55860)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+835971196 ms)
2025-02-17 12:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+835984414 ms)
2025-02-17 12:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+835984415 ms)
2025-02-17 12:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:08:25.400168+00:00', 'runtime': '835980.001', 'adc_vb_f': '12.177', 'adc_va_f': '8.365', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835984416 ms)
2025-02-17 12:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:07:25.400175+00:00', 'runtime': '835920.001', 'adc_vb_f': '12.177', 'adc_va_f': '7.596', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+835984416 ms)
2025-02-17 12:09:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836044416 ms)
2025-02-17 12:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836044416 ms)
2025-02-17 12:09:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:09:25.400175+00:00', 'runtime': '836040.001', 'adc_vb_f': '12.177', 'adc_va_f': '9.222', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836044417 ms)
2025-02-17 12:09:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:08:25.400168+00:00', 'runtime': '835980.001', 'adc_vb_f': '12.177', 'adc_va_f': '8.365', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836044418 ms)
2025-02-17 12:10:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836104416 ms)
2025-02-17 12:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836104416 ms)
2025-02-17 12:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:10:25.400172+00:00', 'runtime': '836100.001', 'adc_vb_f': '12.177', 'adc_va_f': '9.985', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836104417 ms)
2025-02-17 12:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:09:25.400175+00:00', 'runtime': '836040.001', 'adc_vb_f': '12.177', 'adc_va_f': '9.222', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836104417 ms)
2025-02-17 12:10:35.930 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+836113749 ms)
2025-02-17 12:10:35.930 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+836113750 ms)
2025-02-17 12:10:35.938 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:10:25.400342+00:00'} (T+836113757 ms)
2025-02-17 12:11:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836164415 ms)
2025-02-17 12:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836164415 ms)
2025-02-17 12:11:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:11:25.400182+00:00', 'runtime': '836160.001', 'adc_vb_f': '12.177', 'adc_va_f': '10.941', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836164416 ms)
2025-02-17 12:11:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:10:25.400172+00:00', 'runtime': '836100.001', 'adc_vb_f': '12.177', 'adc_va_f': '9.985', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836164417 ms)
2025-02-17 12:12:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836224412 ms)
2025-02-17 12:12:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836224413 ms)
2025-02-17 12:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:12:25.400170+00:00', 'runtime': '836220.001', 'adc_vb_f': '12.177', 'adc_va_f': '11.566', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836224414 ms)
2025-02-17 12:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:11:25.400182+00:00', 'runtime': '836160.001', 'adc_vb_f': '12.177', 'adc_va_f': '10.941', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836224414 ms)
2025-02-17 12:13:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836284412 ms)
2025-02-17 12:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836284412 ms)
2025-02-17 12:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:13:25.400156+00:00', 'runtime': '836280.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.225', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836284413 ms)
2025-02-17 12:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:12:25.400170+00:00', 'runtime': '836220.001', 'adc_vb_f': '12.177', 'adc_va_f': '11.566', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836284414 ms)
2025-02-17 12:13:33.844 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+836291663 ms)
2025-02-17 12:13:33.845 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+836291664 ms)
2025-02-17 12:13:33.845 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56198)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+836291665 ms)
2025-02-17 12:14:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836344413 ms)
2025-02-17 12:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836344414 ms)
2025-02-17 12:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:14:25.400501+00:00', 'runtime': '836340.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.674', 'adc_vl_f': '12.164', 'adc_ic_f': '0.0', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.173', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.891', 'sweep_pmax': '3.276', 'sweep_voc': '13.184', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836344414 ms)
2025-02-17 12:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:13:25.400156+00:00', 'runtime': '836280.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.225', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '0.0', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '8.456', 'sweep_pmax': '3.276', 'sweep_voc': '11.276', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836344415 ms)
2025-02-17 12:15:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836404411 ms)
2025-02-17 12:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836404411 ms)
2025-02-17 12:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:15:25.400173+00:00', 'runtime': '836400.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.762', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.211', 'sweep_pmax': '3.276', 'sweep_voc': '13.611', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836404412 ms)
2025-02-17 12:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:14:25.400501+00:00', 'runtime': '836340.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.674', 'adc_vl_f': '12.164', 'adc_ic_f': '0.0', 'adc_il_f': '0.502', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.173', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.891', 'sweep_pmax': '3.276', 'sweep_voc': '13.184', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836404413 ms)
2025-02-17 12:16:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836464412 ms)
2025-02-17 12:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836464412 ms)
2025-02-17 12:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:16:25.400181+00:00', 'runtime': '836460.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.762', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '10.797', 'sweep_pmax': '3.276', 'sweep_voc': '14.392', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836464413 ms)
2025-02-17 12:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:15:25.400173+00:00', 'runtime': '836400.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.762', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.211', 'sweep_pmax': '3.276', 'sweep_voc': '13.611', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836464414 ms)
2025-02-17 12:17:24.206 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+836522025 ms)
2025-02-17 12:17:24.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+836522025 ms)
2025-02-17 12:17:24.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56452)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+836522026 ms)
2025-02-17 12:17:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836524410 ms)
2025-02-17 12:17:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836524410 ms)
2025-02-17 12:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:17:25.400165+00:00', 'runtime': '836520.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.811', 'adc_vl_f': '12.164', 'adc_ic_f': '0.007', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '11.459', 'sweep_pmax': '3.276', 'sweep_voc': '15.277', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836524411 ms)
2025-02-17 12:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:16:25.400181+00:00', 'runtime': '836460.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.762', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '10.797', 'sweep_pmax': '3.276', 'sweep_voc': '14.392', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836524412 ms)
2025-02-17 12:18:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836584409 ms)
2025-02-17 12:18:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836584409 ms)
2025-02-17 12:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:18:25.400239+00:00', 'runtime': '836580.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.845', 'adc_vl_f': '12.164', 'adc_ic_f': '0.01', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.106', 'sweep_vmp': '11.972', 'sweep_pmax': '3.276', 'sweep_voc': '15.961', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836584410 ms)
2025-02-17 12:18:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:17:25.400165+00:00', 'runtime': '836520.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.811', 'adc_vl_f': '12.164', 'adc_ic_f': '0.007', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '11.459', 'sweep_pmax': '3.276', 'sweep_voc': '15.277', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836584411 ms)
2025-02-17 12:18:30.288 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+836588107 ms)
2025-02-17 12:18:30.288 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+836588107 ms)
2025-02-17 12:18:30.288 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+836588108 ms)
2025-02-17 12:18:51.379 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+836609198 ms)
2025-02-17 12:18:51.379 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+836609198 ms)
2025-02-17 12:18:51.380 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56544)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+836609199 ms)
2025-02-17 12:19:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836644408 ms)
2025-02-17 12:19:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836644409 ms)
2025-02-17 12:19:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:19:25.400160+00:00', 'runtime': '836640.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.866', 'adc_vl_f': '12.164', 'adc_ic_f': '0.014', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.136', 'sweep_vmp': '12.225', 'sweep_pmax': '3.276', 'sweep_voc': '16.296', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836644410 ms)
2025-02-17 12:19:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:18:25.400239+00:00', 'runtime': '836580.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.845', 'adc_vl_f': '12.164', 'adc_ic_f': '0.01', 'adc_il_f': '0.481', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.8', 'ahl_t': '1589.8', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.106', 'sweep_vmp': '11.972', 'sweep_pmax': '3.276', 'sweep_voc': '15.961', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836644410 ms)
2025-02-17 12:20:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836704406 ms)
2025-02-17 12:20:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836704406 ms)
2025-02-17 12:20:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:20:25.400179+00:00', 'runtime': '836700.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.869', 'adc_vl_f': '12.164', 'adc_ic_f': '0.014', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.136', 'sweep_vmp': '12.463', 'sweep_pmax': '3.276', 'sweep_voc': '16.614', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836704407 ms)
2025-02-17 12:20:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:19:25.400160+00:00', 'runtime': '836640.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.866', 'adc_vl_f': '12.164', 'adc_ic_f': '0.014', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.136', 'sweep_vmp': '12.225', 'sweep_pmax': '3.276', 'sweep_voc': '16.296', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836704408 ms)
2025-02-17 12:21:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836764407 ms)
2025-02-17 12:21:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836764408 ms)
2025-02-17 12:21:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:21:25.400218+00:00', 'runtime': '836760.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.985', 'adc_vl_f': '12.164', 'adc_ic_f': '0.014', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.166', 'sweep_vmp': '12.949', 'sweep_pmax': '3.276', 'sweep_voc': '17.261', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836764409 ms)
2025-02-17 12:21:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:20:25.400179+00:00', 'runtime': '836700.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.869', 'adc_vl_f': '12.164', 'adc_ic_f': '0.014', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.136', 'sweep_vmp': '12.463', 'sweep_pmax': '3.276', 'sweep_voc': '16.614', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836764409 ms)
2025-02-17 12:22:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836824404 ms)
2025-02-17 12:22:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836824404 ms)
2025-02-17 12:22:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:22:25.400198+00:00', 'runtime': '836820.001', 'adc_vb_f': '12.177', 'adc_va_f': '13.62', 'adc_vl_f': '12.155', 'adc_ic_f': '0.002', 'adc_il_f': '0.408', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '13.58', 'sweep_pmax': '3.276', 'sweep_voc': '18.103', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836824405 ms)
2025-02-17 12:22:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:21:25.400218+00:00', 'runtime': '836760.001', 'adc_vb_f': '12.177', 'adc_va_f': '12.985', 'adc_vl_f': '12.164', 'adc_ic_f': '0.014', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.166', 'sweep_vmp': '12.949', 'sweep_pmax': '3.276', 'sweep_voc': '17.261', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836824406 ms)
2025-02-17 12:22:54.774 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+836852594 ms)
2025-02-17 12:22:54.775 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+836852594 ms)
2025-02-17 12:22:54.775 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56800)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+836852595 ms)
2025-02-17 12:23:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836884419 ms)
2025-02-17 12:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836884420 ms)
2025-02-17 12:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:23:25.400201+00:00', 'runtime': '836880.001', 'adc_vb_f': '12.177', 'adc_va_f': '13.864', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.173', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '13.837', 'sweep_pmax': '3.276', 'sweep_voc': '18.445', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836884421 ms)
2025-02-17 12:23:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:22:25.400198+00:00', 'runtime': '836820.001', 'adc_vb_f': '12.177', 'adc_va_f': '13.62', 'adc_vl_f': '12.155', 'adc_ic_f': '0.002', 'adc_il_f': '0.408', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '13.58', 'sweep_pmax': '3.276', 'sweep_voc': '18.103', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836884421 ms)
2025-02-17 12:24:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+836944417 ms)
2025-02-17 12:24:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+836944417 ms)
2025-02-17 12:24:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:24:25.400200+00:00', 'runtime': '836940.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.056', 'adc_vl_f': '12.164', 'adc_ic_f': '0.0', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '14.032', 'sweep_pmax': '3.276', 'sweep_voc': '18.707', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836944418 ms)
2025-02-17 12:24:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:23:25.400201+00:00', 'runtime': '836880.001', 'adc_vb_f': '12.177', 'adc_va_f': '13.864', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.173', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '13.837', 'sweep_pmax': '3.276', 'sweep_voc': '18.445', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+836944419 ms)
2025-02-17 12:25:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837004414 ms)
2025-02-17 12:25:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837004415 ms)
2025-02-17 12:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:25:25.400160+00:00', 'runtime': '837000.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.355', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.322', 'sweep_pmax': '3.276', 'sweep_voc': '19.092', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837004416 ms)
2025-02-17 12:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:24:25.400200+00:00', 'runtime': '836940.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.056', 'adc_vl_f': '12.164', 'adc_ic_f': '0.0', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '14.032', 'sweep_pmax': '3.276', 'sweep_voc': '18.707', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837004416 ms)
2025-02-17 12:26:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837064415 ms)
2025-02-17 12:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837064415 ms)
2025-02-17 12:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:26:25.400161+00:00', 'runtime': '837060.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.493', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837064416 ms)
2025-02-17 12:26:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:25:25.400160+00:00', 'runtime': '837000.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.355', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.322', 'sweep_pmax': '3.276', 'sweep_voc': '19.092', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837064417 ms)
2025-02-17 12:27:24.196 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+837122015 ms)
2025-02-17 12:27:24.196 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+837122016 ms)
2025-02-17 12:27:24.197 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57092)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+837122016 ms)
2025-02-17 12:27:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837124414 ms)
2025-02-17 12:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837124415 ms)
2025-02-17 12:27:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:27:25.400160+00:00', 'runtime': '837120.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.496', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837124416 ms)
2025-02-17 12:27:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:26:25.400161+00:00', 'runtime': '837060.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.493', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837124416 ms)
2025-02-17 12:28:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837184411 ms)
2025-02-17 12:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837184411 ms)
2025-02-17 12:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:28:25.400195+00:00', 'runtime': '837180.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.481', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837184412 ms)
2025-02-17 12:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:27:25.400160+00:00', 'runtime': '837120.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.496', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837184413 ms)
2025-02-17 12:28:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837193433 ms)
2025-02-17 12:28:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837193434 ms)
2025-02-17 12:28:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:28:25.400341+00:00'} (T+837193435 ms)
2025-02-17 12:29:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837244411 ms)
2025-02-17 12:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837244411 ms)
2025-02-17 12:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:29:25.400265+00:00', 'runtime': '837240.001', 'adc_vb_f': '12.173', 'adc_va_f': '14.496', 'adc_vl_f': '12.161', 'adc_ic_f': '0.002', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.173', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837244412 ms)
2025-02-17 12:29:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:28:25.400195+00:00', 'runtime': '837180.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.481', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837244413 ms)
2025-02-17 12:29:37.102 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837254922 ms)
2025-02-17 12:29:37.103 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837254922 ms)
2025-02-17 12:29:37.104 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:29:25.400272+00:00'} (T+837254923 ms)
2025-02-17 12:30:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837304410 ms)
2025-02-17 12:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837304410 ms)
2025-02-17 12:30:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:30:25.400161+00:00', 'runtime': '837300.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.496', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837304414 ms)
2025-02-17 12:30:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:29:25.400265+00:00', 'runtime': '837240.001', 'adc_vb_f': '12.173', 'adc_va_f': '14.496', 'adc_vl_f': '12.161', 'adc_ic_f': '0.002', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.173', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837304415 ms)
2025-02-17 12:31:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837364408 ms)
2025-02-17 12:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837364408 ms)
2025-02-17 12:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:31:25.400229+00:00', 'runtime': '837360.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.493', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837364413 ms)
2025-02-17 12:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:30:25.400161+00:00', 'runtime': '837300.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.496', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837364413 ms)
2025-02-17 12:32:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837424407 ms)
2025-02-17 12:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837424407 ms)
2025-02-17 12:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:32:25.400197+00:00', 'runtime': '837420.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.603', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.56', 'sweep_pmax': '3.276', 'sweep_voc': '19.409', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837424411 ms)
2025-02-17 12:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:31:25.400229+00:00', 'runtime': '837360.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.493', 'adc_vl_f': '12.164', 'adc_ic_f': '0.002', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1589.9', 'ahl_t': '1589.9', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '3.276', 'sweep_voc': '19.299', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837424412 ms)
2025-02-17 12:33:06.632 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+837464451 ms)
2025-02-17 12:33:06.632 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+837464452 ms)
2025-02-17 12:33:06.633 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+837464452 ms)
2025-02-17 12:33:06.634 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+837464453 ms)
2025-02-17 12:33:06.634 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+837464454 ms)
2025-02-17 12:33:06.635 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+837464454 ms)
2025-02-17 12:33:06.636 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+837464455 ms)
2025-02-17 12:33:06.636 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+837464455 ms)
2025-02-17 12:33:06.637 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+837464456 ms)
2025-02-17 12:33:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837484403 ms)
2025-02-17 12:33:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837484404 ms)
2025-02-17 12:33:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:33:25.400180+00:00', 'runtime': '837480.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.749', 'adc_vl_f': '12.164', 'adc_ic_f': '0.007', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.075', 'sweep_vmp': '14.716', 'sweep_pmax': '3.276', 'sweep_voc': '19.617', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837484405 ms)
2025-02-17 12:33:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:32:25.400197+00:00', 'runtime': '837420.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.603', 'adc_vl_f': '12.164', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.56', 'sweep_pmax': '3.276', 'sweep_voc': '19.409', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837484406 ms)
2025-02-17 12:34:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837544402 ms)
2025-02-17 12:34:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837544402 ms)
2025-02-17 12:34:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:34:25.400179+00:00', 'runtime': '837540.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.841', 'adc_vl_f': '12.164', 'adc_ic_f': '0.012', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.106', 'sweep_vmp': '14.841', 'sweep_pmax': '3.276', 'sweep_voc': '19.785', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837544404 ms)
2025-02-17 12:34:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:33:25.400180+00:00', 'runtime': '837480.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.749', 'adc_vl_f': '12.164', 'adc_ic_f': '0.007', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.075', 'sweep_vmp': '14.716', 'sweep_pmax': '3.276', 'sweep_voc': '19.617', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837544405 ms)
2025-02-17 12:35:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837604418 ms)
2025-02-17 12:35:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837604419 ms)
2025-02-17 12:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:35:25.400269+00:00', 'runtime': '837600.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.987', 'adc_vl_f': '12.164', 'adc_ic_f': '0.022', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.226', 'sweep_vmp': '14.975', 'sweep_pmax': '3.276', 'sweep_voc': '19.965', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837604421 ms)
2025-02-17 12:35:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:34:25.400179+00:00', 'runtime': '837540.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.841', 'adc_vl_f': '12.164', 'adc_ic_f': '0.012', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.106', 'sweep_vmp': '14.841', 'sweep_pmax': '3.276', 'sweep_voc': '19.785', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837604421 ms)
2025-02-17 12:35:35.728 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837613547 ms)
2025-02-17 12:35:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837613548 ms)
2025-02-17 12:35:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:35:25.400346+00:00'} (T+837613549 ms)
2025-02-17 12:36:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837664417 ms)
2025-02-17 12:36:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837664418 ms)
2025-02-17 12:36:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:36:25.400190+00:00', 'runtime': '837660.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.14', 'adc_vl_f': '12.164', 'adc_ic_f': '0.029', 'adc_il_f': '0.507', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.347', 'sweep_vmp': '15.118', 'sweep_pmax': '3.276', 'sweep_voc': '20.154', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837664418 ms)
2025-02-17 12:36:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:35:25.400269+00:00', 'runtime': '837600.001', 'adc_vb_f': '12.177', 'adc_va_f': '14.987', 'adc_vl_f': '12.164', 'adc_ic_f': '0.022', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.226', 'sweep_vmp': '14.975', 'sweep_pmax': '3.276', 'sweep_voc': '19.965', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837664419 ms)
2025-02-17 12:36:29.977 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+837667797 ms)
2025-02-17 12:36:29.978 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+837667797 ms)
2025-02-17 12:36:29.978 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57676)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+837667797 ms)
2025-02-17 12:37:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837724417 ms)
2025-02-17 12:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837724418 ms)
2025-02-17 12:37:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:37:25.400158+00:00', 'runtime': '837720.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.231', 'adc_vl_f': '12.164', 'adc_ic_f': '0.046', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.559', 'sweep_vmp': '15.201', 'sweep_pmax': '3.276', 'sweep_voc': '20.264', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837724418 ms)
2025-02-17 12:37:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:36:25.400190+00:00', 'runtime': '837660.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.14', 'adc_vl_f': '12.164', 'adc_ic_f': '0.029', 'adc_il_f': '0.507', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.347', 'sweep_vmp': '15.118', 'sweep_pmax': '3.276', 'sweep_voc': '20.154', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837724421 ms)
2025-02-17 12:38:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837784413 ms)
2025-02-17 12:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837784413 ms)
2025-02-17 12:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:38:25.400172+00:00', 'runtime': '837780.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.329', 'adc_vl_f': '12.164', 'adc_ic_f': '0.058', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.664', 'sweep_vmp': '15.283', 'sweep_pmax': '3.276', 'sweep_voc': '20.374', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837784414 ms)
2025-02-17 12:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:37:25.400158+00:00', 'runtime': '837720.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.231', 'adc_vl_f': '12.164', 'adc_ic_f': '0.046', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.559', 'sweep_vmp': '15.201', 'sweep_pmax': '3.276', 'sweep_voc': '20.264', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837784417 ms)
2025-02-17 12:38:35.676 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837793495 ms)
2025-02-17 12:38:35.676 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837793495 ms)
2025-02-17 12:38:35.677 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:38:25.400322+00:00'} (T+837793496 ms)
2025-02-17 12:39:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837844414 ms)
2025-02-17 12:39:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837844415 ms)
2025-02-17 12:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:39:25.400193+00:00', 'runtime': '837840.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.378', 'adc_vl_f': '12.164', 'adc_ic_f': '0.068', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.815', 'sweep_vmp': '15.366', 'sweep_pmax': '3.276', 'sweep_voc': '20.483', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837844416 ms)
2025-02-17 12:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:38:25.400172+00:00', 'runtime': '837780.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.329', 'adc_vl_f': '12.164', 'adc_ic_f': '0.058', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.664', 'sweep_vmp': '15.283', 'sweep_pmax': '3.276', 'sweep_voc': '20.374', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837844416 ms)
2025-02-17 12:39:35.891 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837853710 ms)
2025-02-17 12:39:35.891 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+837853710 ms)
2025-02-17 12:39:35.893 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:39:25.400142+00:00'} (T+837853712 ms)
2025-02-17 12:40:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837904414 ms)
2025-02-17 12:40:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837904415 ms)
2025-02-17 12:40:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:40:25.400419+00:00', 'runtime': '837900.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.472', 'adc_vl_f': '12.164', 'adc_ic_f': '0.077', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.936', 'sweep_vmp': '15.448', 'sweep_pmax': '3.276', 'sweep_voc': '20.593', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837904416 ms)
2025-02-17 12:40:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:39:25.400193+00:00', 'runtime': '837840.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.378', 'adc_vl_f': '12.164', 'adc_ic_f': '0.068', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.815', 'sweep_vmp': '15.366', 'sweep_pmax': '3.276', 'sweep_voc': '20.483', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837904416 ms)
2025-02-17 12:41:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+837964414 ms)
2025-02-17 12:41:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+837964414 ms)
2025-02-17 12:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:41:25.400175+00:00', 'runtime': '837960.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.524', 'adc_vl_f': '12.164', 'adc_ic_f': '0.104', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.223', 'sweep_vmp': '15.512', 'sweep_pmax': '3.276', 'sweep_voc': '20.679', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837964416 ms)
2025-02-17 12:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:40:25.400419+00:00', 'runtime': '837900.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.472', 'adc_vl_f': '12.164', 'adc_ic_f': '0.077', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.443', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.936', 'sweep_vmp': '15.448', 'sweep_pmax': '3.276', 'sweep_voc': '20.593', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+837964416 ms)
2025-02-17 12:42:12.484 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838010304 ms)
2025-02-17 12:42:12.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838010304 ms)
2025-02-17 12:42:12.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58040)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838010305 ms)
2025-02-17 12:42:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838024413 ms)
2025-02-17 12:42:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838024413 ms)
2025-02-17 12:42:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:42:25.400199+00:00', 'runtime': '838020.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.625', 'adc_vl_f': '12.164', 'adc_ic_f': '0.128', 'adc_il_f': '0.362', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.525', 'sweep_vmp': '15.61', 'sweep_pmax': '3.276', 'sweep_voc': '20.81', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838024414 ms)
2025-02-17 12:42:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:41:25.400175+00:00', 'runtime': '837960.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.524', 'adc_vl_f': '12.164', 'adc_ic_f': '0.104', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.223', 'sweep_vmp': '15.512', 'sweep_pmax': '3.276', 'sweep_voc': '20.679', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838024414 ms)
2025-02-17 12:43:15.645 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838073464 ms)
2025-02-17 12:43:15.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838073465 ms)
2025-02-17 12:43:15.646 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58106)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838073465 ms)
2025-02-17 12:43:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838084410 ms)
2025-02-17 12:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838084410 ms)
2025-02-17 12:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:43:25.400171+00:00', 'runtime': '838080.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.723', 'adc_vl_f': '12.164', 'adc_ic_f': '0.133', 'adc_il_f': '0.343', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.616', 'sweep_vmp': '15.704', 'sweep_pmax': '3.276', 'sweep_voc': '20.938', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838084411 ms)
2025-02-17 12:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:42:25.400199+00:00', 'runtime': '838020.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.625', 'adc_vl_f': '12.164', 'adc_ic_f': '0.128', 'adc_il_f': '0.362', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.525', 'sweep_vmp': '15.61', 'sweep_pmax': '3.276', 'sweep_voc': '20.81', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838084412 ms)
2025-02-17 12:43:44.775 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838102594 ms)
2025-02-17 12:43:44.775 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838102595 ms)
2025-02-17 12:43:44.776 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58138)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838102595 ms)
2025-02-17 12:44:09.896 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838127715 ms)
2025-02-17 12:44:09.896 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838127715 ms)
2025-02-17 12:44:09.897 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58166)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838127716 ms)
2025-02-17 12:44:09.934 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838127753 ms)
2025-02-17 12:44:09.934 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838127753 ms)
2025-02-17 12:44:09.935 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58168)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838127754 ms)
2025-02-17 12:44:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838144409 ms)
2025-02-17 12:44:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838144409 ms)
2025-02-17 12:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:44:25.400162+00:00', 'runtime': '838140.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.765', 'adc_vl_f': '12.164', 'adc_ic_f': '0.133', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.585', 'sweep_vmp': '15.768', 'sweep_pmax': '3.276', 'sweep_voc': '21.021', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838144410 ms)
2025-02-17 12:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:43:25.400171+00:00', 'runtime': '838080.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.723', 'adc_vl_f': '12.164', 'adc_ic_f': '0.133', 'adc_il_f': '0.343', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.616', 'sweep_vmp': '15.704', 'sweep_pmax': '3.276', 'sweep_voc': '20.938', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838144411 ms)
2025-02-17 12:45:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838204408 ms)
2025-02-17 12:45:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838204408 ms)
2025-02-17 12:45:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:45:25.400173+00:00', 'runtime': '838200.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.723', 'adc_vl_f': '12.164', 'adc_ic_f': '0.123', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.495', 'sweep_vmp': '15.698', 'sweep_pmax': '3.276', 'sweep_voc': '20.929', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838204409 ms)
2025-02-17 12:45:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:44:25.400162+00:00', 'runtime': '838140.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.765', 'adc_vl_f': '12.164', 'adc_ic_f': '0.133', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.585', 'sweep_vmp': '15.768', 'sweep_pmax': '3.276', 'sweep_voc': '21.021', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838204409 ms)
2025-02-17 12:46:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838264406 ms)
2025-02-17 12:46:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838264407 ms)
2025-02-17 12:46:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:46:25.400470+00:00', 'runtime': '838260.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.726', 'adc_vl_f': '12.164', 'adc_ic_f': '0.126', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.555', 'sweep_vmp': '15.686', 'sweep_pmax': '3.276', 'sweep_voc': '20.911', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838264408 ms)
2025-02-17 12:46:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:45:25.400173+00:00', 'runtime': '838200.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.723', 'adc_vl_f': '12.164', 'adc_ic_f': '0.123', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.0', 'ahl_t': '1590.0', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.495', 'sweep_vmp': '15.698', 'sweep_pmax': '3.276', 'sweep_voc': '20.929', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838264408 ms)
2025-02-17 12:46:50.211 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838288030 ms)
2025-02-17 12:46:50.211 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838288030 ms)
2025-02-17 12:46:50.211 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58348)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838288031 ms)
2025-02-17 12:47:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838324420 ms)
2025-02-17 12:47:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838324420 ms)
2025-02-17 12:47:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:47:25.400195+00:00', 'runtime': '838320.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.765', 'adc_vl_f': '12.164', 'adc_ic_f': '0.143', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.721', 'sweep_vmp': '15.75', 'sweep_pmax': '3.276', 'sweep_voc': '20.996', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838324422 ms)
2025-02-17 12:47:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:46:25.400470+00:00', 'runtime': '838260.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.726', 'adc_vl_f': '12.164', 'adc_ic_f': '0.126', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.555', 'sweep_vmp': '15.686', 'sweep_pmax': '3.276', 'sweep_voc': '20.911', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838324422 ms)
2025-02-17 12:47:35.423 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838333242 ms)
2025-02-17 12:47:35.424 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838333243 ms)
2025-02-17 12:47:35.425 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:47:25.413046+00:00'} (T+838333244 ms)
2025-02-17 12:47:49.369 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838347189 ms)
2025-02-17 12:47:49.370 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838347189 ms)
2025-02-17 12:47:49.370 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58410)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838347190 ms)
2025-02-17 12:48:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838384418 ms)
2025-02-17 12:48:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838384419 ms)
2025-02-17 12:48:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:48:25.400162+00:00', 'runtime': '838380.001', 'adc_vb_f': '12.18', 'adc_va_f': '15.872', 'adc_vl_f': '12.164', 'adc_ic_f': '0.167', 'adc_il_f': '0.324', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.053', 'sweep_vmp': '15.826', 'sweep_pmax': '3.276', 'sweep_voc': '21.1', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838384420 ms)
2025-02-17 12:48:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:47:25.400195+00:00', 'runtime': '838320.001', 'adc_vb_f': '12.177', 'adc_va_f': '15.765', 'adc_vl_f': '12.164', 'adc_ic_f': '0.143', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.721', 'sweep_vmp': '15.75', 'sweep_pmax': '3.276', 'sweep_voc': '20.996', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838384420 ms)
2025-02-17 12:48:35.851 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838393671 ms)
2025-02-17 12:48:35.852 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838393671 ms)
2025-02-17 12:48:35.853 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:48:25.400368+00:00'} (T+838393672 ms)
2025-02-17 12:48:51.493 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838409312 ms)
2025-02-17 12:48:51.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838409313 ms)
2025-02-17 12:48:51.494 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58480)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838409313 ms)
2025-02-17 12:49:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838444416 ms)
2025-02-17 12:49:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838444416 ms)
2025-02-17 12:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:49:25.400154+00:00', 'runtime': '838440.001', 'adc_vb_f': '12.192', 'adc_va_f': '15.964', 'adc_vl_f': '12.164', 'adc_ic_f': '0.193', 'adc_il_f': '0.355', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.31', 'sweep_vmp': '15.924', 'sweep_pmax': '3.276', 'sweep_voc': '21.228', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838444417 ms)
2025-02-17 12:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:48:25.400162+00:00', 'runtime': '838380.001', 'adc_vb_f': '12.18', 'adc_va_f': '15.872', 'adc_vl_f': '12.164', 'adc_ic_f': '0.167', 'adc_il_f': '0.324', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.177', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.053', 'sweep_vmp': '15.826', 'sweep_pmax': '3.276', 'sweep_voc': '21.1', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838444418 ms)
2025-02-17 12:49:35.910 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838453730 ms)
2025-02-17 12:49:35.911 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838453730 ms)
2025-02-17 12:49:35.912 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:49:25.400452+00:00'} (T+838453731 ms)
2025-02-17 12:50:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838504414 ms)
2025-02-17 12:50:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838504415 ms)
2025-02-17 12:50:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:50:25.402806+00:00', 'runtime': '838500.003', 'adc_vb_f': '12.207', 'adc_va_f': '16.013', 'adc_vl_f': '12.164', 'adc_ic_f': '0.213', 'adc_il_f': '0.329', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.552', 'sweep_vmp': '16.006', 'sweep_pmax': '3.276', 'sweep_voc': '21.338', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838504419 ms)
2025-02-17 12:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:49:25.400154+00:00', 'runtime': '838440.001', 'adc_vb_f': '12.192', 'adc_va_f': '15.964', 'adc_vl_f': '12.164', 'adc_ic_f': '0.193', 'adc_il_f': '0.355', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.183', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.31', 'sweep_vmp': '15.924', 'sweep_pmax': '3.276', 'sweep_voc': '21.228', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838504419 ms)
2025-02-17 12:50:35.871 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838513690 ms)
2025-02-17 12:50:35.871 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838513691 ms)
2025-02-17 12:50:35.872 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:50:25.400199+00:00'} (T+838513691 ms)
2025-02-17 12:51:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838564412 ms)
2025-02-17 12:51:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838564413 ms)
2025-02-17 12:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:51:25.402782+00:00', 'runtime': '838560.003', 'adc_vb_f': '12.207', 'adc_va_f': '16.016', 'adc_vl_f': '12.164', 'adc_ic_f': '0.215', 'adc_il_f': '0.329', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.612', 'sweep_vmp': '16.013', 'sweep_pmax': '3.276', 'sweep_voc': '21.347', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838564417 ms)
2025-02-17 12:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:50:25.402806+00:00', 'runtime': '838500.003', 'adc_vb_f': '12.207', 'adc_va_f': '16.013', 'adc_vl_f': '12.164', 'adc_ic_f': '0.213', 'adc_il_f': '0.329', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.195', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.552', 'sweep_vmp': '16.006', 'sweep_pmax': '3.276', 'sweep_voc': '21.338', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838564418 ms)
2025-02-17 12:51:31.751 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838569570 ms)
2025-02-17 12:51:31.751 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838569570 ms)
2025-02-17 12:51:31.752 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58650)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838569571 ms)
2025-02-17 12:51:58.897 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838596716 ms)
2025-02-17 12:51:58.897 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838596716 ms)
2025-02-17 12:51:58.897 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58682)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838596717 ms)
2025-02-17 12:52:15.002 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838612821 ms)
2025-02-17 12:52:15.002 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838612821 ms)
2025-02-17 12:52:15.002 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58700)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838612822 ms)
2025-02-17 12:52:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838624411 ms)
2025-02-17 12:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838624411 ms)
2025-02-17 12:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:52:25.400163+00:00', 'runtime': '838620.001', 'adc_vb_f': '12.207', 'adc_va_f': '16.068', 'adc_vl_f': '12.164', 'adc_ic_f': '0.217', 'adc_il_f': '0.374', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.642', 'sweep_vmp': '16.022', 'sweep_pmax': '3.276', 'sweep_voc': '21.359', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838624413 ms)
2025-02-17 12:52:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:51:25.402782+00:00', 'runtime': '838560.003', 'adc_vb_f': '12.207', 'adc_va_f': '16.016', 'adc_vl_f': '12.164', 'adc_ic_f': '0.215', 'adc_il_f': '0.329', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.201', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.612', 'sweep_vmp': '16.013', 'sweep_pmax': '3.276', 'sweep_voc': '21.347', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838624414 ms)
2025-02-17 12:53:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838684408 ms)
2025-02-17 12:53:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838684409 ms)
2025-02-17 12:53:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:53:25.400188+00:00', 'runtime': '838680.001', 'adc_vb_f': '12.21', 'adc_va_f': '16.064', 'adc_vl_f': '12.164', 'adc_ic_f': '0.222', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.703', 'sweep_vmp': '16.019', 'sweep_pmax': '3.276', 'sweep_voc': '21.356', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838684411 ms)
2025-02-17 12:53:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:52:25.400163+00:00', 'runtime': '838620.001', 'adc_vb_f': '12.207', 'adc_va_f': '16.068', 'adc_vl_f': '12.164', 'adc_ic_f': '0.217', 'adc_il_f': '0.374', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.204', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.642', 'sweep_vmp': '16.022', 'sweep_pmax': '3.276', 'sweep_voc': '21.359', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838684411 ms)
2025-02-17 12:54:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838744407 ms)
2025-02-17 12:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838744408 ms)
2025-02-17 12:54:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:54:25.400199+00:00', 'runtime': '838740.001', 'adc_vb_f': '12.216', 'adc_va_f': '16.116', 'adc_vl_f': '12.164', 'adc_ic_f': '0.244', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.944', 'sweep_vmp': '16.089', 'sweep_pmax': '3.276', 'sweep_voc': '21.448', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838744409 ms)
2025-02-17 12:54:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:53:25.400188+00:00', 'runtime': '838680.001', 'adc_vb_f': '12.21', 'adc_va_f': '16.064', 'adc_vl_f': '12.164', 'adc_ic_f': '0.222', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.207', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20119', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.703', 'sweep_vmp': '16.019', 'sweep_pmax': '3.276', 'sweep_voc': '21.356', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838744410 ms)
2025-02-17 12:55:22.310 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838800129 ms)
2025-02-17 12:55:22.310 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838800129 ms)
2025-02-17 12:55:22.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58896)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838800130 ms)
2025-02-17 12:55:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838804404 ms)
2025-02-17 12:55:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838804405 ms)
2025-02-17 12:55:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:55:25.400647+00:00', 'runtime': '838800.001', 'adc_vb_f': '12.216', 'adc_va_f': '18.069', 'adc_vl_f': '12.164', 'adc_ic_f': '0.278', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.352', 'sweep_vmp': '18.066', 'sweep_pmax': '3.88', 'sweep_voc': '21.558', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838804406 ms)
2025-02-17 12:55:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:54:25.400199+00:00', 'runtime': '838740.001', 'adc_vb_f': '12.216', 'adc_va_f': '16.116', 'adc_vl_f': '12.164', 'adc_ic_f': '0.244', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.944', 'sweep_vmp': '16.089', 'sweep_pmax': '3.276', 'sweep_voc': '21.448', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838804408 ms)
2025-02-17 12:55:41.397 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838819216 ms)
2025-02-17 12:55:41.397 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838819216 ms)
2025-02-17 12:55:41.398 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58916)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838819217 ms)
2025-02-17 12:55:56.511 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+838834330 ms)
2025-02-17 12:55:56.511 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+838834330 ms)
2025-02-17 12:55:56.512 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58944)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+838834331 ms)
2025-02-17 12:56:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838864420 ms)
2025-02-17 12:56:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838864421 ms)
2025-02-17 12:56:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:56:25.402776+00:00', 'runtime': '838860.003', 'adc_vb_f': '12.225', 'adc_va_f': '18.607', 'adc_vl_f': '12.186', 'adc_ic_f': '0.319', 'adc_il_f': '0.382', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.865', 'sweep_vmp': '18.622', 'sweep_pmax': '3.88', 'sweep_voc': '21.643', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838864422 ms)
2025-02-17 12:56:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:55:25.400647+00:00', 'runtime': '838800.001', 'adc_vb_f': '12.216', 'adc_va_f': '18.069', 'adc_vl_f': '12.164', 'adc_ic_f': '0.278', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.213', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.352', 'sweep_vmp': '18.066', 'sweep_pmax': '3.88', 'sweep_voc': '21.558', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838864423 ms)
2025-02-17 12:56:35.881 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838873700 ms)
2025-02-17 12:56:35.881 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838873701 ms)
2025-02-17 12:56:35.882 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:56:25.400145+00:00'} (T+838873702 ms)
2025-02-17 12:57:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838924408 ms)
2025-02-17 12:57:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838924408 ms)
2025-02-17 12:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:57:25.400147+00:00', 'runtime': '838920.001', 'adc_vb_f': '12.247', 'adc_va_f': '18.604', 'adc_vl_f': '12.204', 'adc_ic_f': '0.35', 'adc_il_f': '0.358', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.258', 'sweep_vmp': '18.622', 'sweep_pmax': '3.88', 'sweep_voc': '21.643', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838924409 ms)
2025-02-17 12:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:56:25.402776+00:00', 'runtime': '838860.003', 'adc_vb_f': '12.225', 'adc_va_f': '18.607', 'adc_vl_f': '12.186', 'adc_ic_f': '0.319', 'adc_il_f': '0.382', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.219', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.865', 'sweep_vmp': '18.622', 'sweep_pmax': '3.88', 'sweep_voc': '21.643', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838924410 ms)
2025-02-17 12:57:35.988 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838933807 ms)
2025-02-17 12:57:35.989 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838933808 ms)
2025-02-17 12:57:35.990 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:57:25.400426+00:00'} (T+838933809 ms)
2025-02-17 12:58:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+838984409 ms)
2025-02-17 12:58:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+838984409 ms)
2025-02-17 12:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:58:25.400191+00:00', 'runtime': '838980.001', 'adc_vb_f': '12.253', 'adc_va_f': '18.604', 'adc_vl_f': '12.204', 'adc_ic_f': '0.384', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.696', 'sweep_vmp': '18.622', 'sweep_pmax': '3.88', 'sweep_voc': '21.643', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838984410 ms)
2025-02-17 12:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:57:25.400147+00:00', 'runtime': '838920.001', 'adc_vb_f': '12.247', 'adc_va_f': '18.604', 'adc_vl_f': '12.204', 'adc_ic_f': '0.35', 'adc_il_f': '0.358', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.238', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.258', 'sweep_vmp': '18.622', 'sweep_pmax': '3.88', 'sweep_voc': '21.643', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+838984410 ms)
2025-02-17 12:58:35.757 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838993576 ms)
2025-02-17 12:58:35.757 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+838993577 ms)
2025-02-17 12:58:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 12:58:25.400342+00:00'} (T+838993578 ms)
2025-02-17 12:59:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839044407 ms)
2025-02-17 12:59:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839044408 ms)
2025-02-17 12:59:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 12:59:25.400213+00:00', 'runtime': '839040.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.199', 'adc_vl_f': '12.204', 'adc_ic_f': '0.418', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.088', 'sweep_vmp': '19.199', 'sweep_pmax': '5.088', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839044409 ms)
2025-02-17 12:59:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:58:25.400191+00:00', 'runtime': '838980.001', 'adc_vb_f': '12.253', 'adc_va_f': '18.604', 'adc_vl_f': '12.204', 'adc_ic_f': '0.384', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.247', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.696', 'sweep_vmp': '18.622', 'sweep_pmax': '3.88', 'sweep_voc': '21.643', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839044409 ms)
2025-02-17 13:00:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839104410 ms)
2025-02-17 13:00:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839104410 ms)
2025-02-17 13:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:00:25.400174+00:00', 'runtime': '839100.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.199', 'adc_vl_f': '12.219', 'adc_ic_f': '0.432', 'adc_il_f': '0.377', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.285', 'sweep_vmp': '19.199', 'sweep_pmax': '5.088', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839104411 ms)
2025-02-17 13:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 12:59:25.400213+00:00', 'runtime': '839040.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.199', 'adc_vl_f': '12.204', 'adc_ic_f': '0.418', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.25', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.088', 'sweep_vmp': '19.199', 'sweep_pmax': '5.088', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839104412 ms)
2025-02-17 13:01:20.996 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+839158815 ms)
2025-02-17 13:01:20.996 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+839158816 ms)
2025-02-17 13:01:20.997 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59288)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+839158816 ms)
2025-02-17 13:01:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839164411 ms)
2025-02-17 13:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839164411 ms)
2025-02-17 13:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:01:25.400164+00:00', 'runtime': '839160.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.193', 'adc_vl_f': '12.234', 'adc_ic_f': '0.452', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.496', 'sweep_vmp': '19.199', 'sweep_pmax': '5.088', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839164412 ms)
2025-02-17 13:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:00:25.400174+00:00', 'runtime': '839100.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.199', 'adc_vl_f': '12.219', 'adc_ic_f': '0.432', 'adc_il_f': '0.377', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.285', 'sweep_vmp': '19.199', 'sweep_pmax': '5.088', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839164413 ms)
2025-02-17 13:01:35.896 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839173715 ms)
2025-02-17 13:01:35.897 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839173716 ms)
2025-02-17 13:01:35.898 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:01:25.400327+00:00'} (T+839173717 ms)
2025-02-17 13:01:42.110 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+839179929 ms)
2025-02-17 13:01:42.110 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+839179929 ms)
2025-02-17 13:01:42.111 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59312)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+839179930 ms)
2025-02-17 13:02:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839224412 ms)
2025-02-17 13:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839224412 ms)
2025-02-17 13:02:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:02:25.400198+00:00', 'runtime': '839220.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.241', 'adc_vl_f': '12.238', 'adc_ic_f': '0.466', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.722', 'sweep_vmp': '19.269', 'sweep_pmax': '5.783', 'sweep_voc': '21.985', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839224416 ms)
2025-02-17 13:02:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:01:25.400164+00:00', 'runtime': '839160.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.193', 'adc_vl_f': '12.234', 'adc_ic_f': '0.452', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.1', 'ahl_t': '1590.1', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.496', 'sweep_vmp': '19.199', 'sweep_pmax': '5.088', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839224417 ms)
2025-02-17 13:02:28.169 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+839225989 ms)
2025-02-17 13:02:28.170 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+839225989 ms)
2025-02-17 13:02:28.170 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+839225990 ms)
2025-02-17 13:03:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839284411 ms)
2025-02-17 13:03:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839284412 ms)
2025-02-17 13:03:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:03:25.400201+00:00', 'runtime': '839280.001', 'adc_vb_f': '12.265', 'adc_va_f': '19.244', 'adc_vl_f': '12.241', 'adc_ic_f': '0.481', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.262', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.858', 'sweep_vmp': '19.269', 'sweep_pmax': '5.783', 'sweep_voc': '21.985', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839284413 ms)
2025-02-17 13:03:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:02:25.400198+00:00', 'runtime': '839220.001', 'adc_vb_f': '12.253', 'adc_va_f': '19.241', 'adc_vl_f': '12.238', 'adc_ic_f': '0.466', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.253', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.722', 'sweep_vmp': '19.269', 'sweep_pmax': '5.783', 'sweep_voc': '21.985', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839284413 ms)
2025-02-17 13:03:50.331 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+839308150 ms)
2025-02-17 13:03:50.331 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+839308151 ms)
2025-02-17 13:03:50.332 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59448)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+839308151 ms)
2025-02-17 13:04:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839344410 ms)
2025-02-17 13:04:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839344410 ms)
2025-02-17 13:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:04:25.400208+00:00', 'runtime': '839340.001', 'adc_vb_f': '12.283', 'adc_va_f': '19.244', 'adc_vl_f': '12.241', 'adc_ic_f': '0.493', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.039', 'sweep_vmp': '19.269', 'sweep_pmax': '5.783', 'sweep_voc': '21.985', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839344411 ms)
2025-02-17 13:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:03:25.400201+00:00', 'runtime': '839280.001', 'adc_vb_f': '12.265', 'adc_va_f': '19.244', 'adc_vl_f': '12.241', 'adc_ic_f': '0.481', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.262', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.858', 'sweep_vmp': '19.269', 'sweep_pmax': '5.783', 'sweep_voc': '21.985', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839344412 ms)
2025-02-17 13:05:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839404410 ms)
2025-02-17 13:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839404410 ms)
2025-02-17 13:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:05:25.400229+00:00', 'runtime': '839400.001', 'adc_vb_f': '12.292', 'adc_va_f': '19.635', 'adc_vl_f': '12.241', 'adc_ic_f': '0.505', 'adc_il_f': '0.389', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.19', 'sweep_vmp': '19.656', 'sweep_pmax': '6.311', 'sweep_voc': '22.043', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839404411 ms)
2025-02-17 13:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:04:25.400208+00:00', 'runtime': '839340.001', 'adc_vb_f': '12.283', 'adc_va_f': '19.244', 'adc_vl_f': '12.241', 'adc_ic_f': '0.493', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.274', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.039', 'sweep_vmp': '19.269', 'sweep_pmax': '5.783', 'sweep_voc': '21.985', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839404412 ms)
2025-02-17 13:05:35.824 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839413643 ms)
2025-02-17 13:05:35.824 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839413644 ms)
2025-02-17 13:05:35.826 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:05:25.400337+00:00'} (T+839413645 ms)
2025-02-17 13:06:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839464410 ms)
2025-02-17 13:06:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839464411 ms)
2025-02-17 13:06:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:06:25.402591+00:00', 'runtime': '839460.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.635', 'adc_vl_f': '12.253', 'adc_ic_f': '0.519', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.341', 'sweep_vmp': '19.656', 'sweep_pmax': '6.311', 'sweep_voc': '22.043', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839464412 ms)
2025-02-17 13:06:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:05:25.400229+00:00', 'runtime': '839400.001', 'adc_vb_f': '12.292', 'adc_va_f': '19.635', 'adc_vl_f': '12.241', 'adc_ic_f': '0.505', 'adc_il_f': '0.389', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.283', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.19', 'sweep_vmp': '19.656', 'sweep_pmax': '6.311', 'sweep_voc': '22.043', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839464412 ms)
2025-02-17 13:06:35.747 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839473566 ms)
2025-02-17 13:06:35.747 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839473567 ms)
2025-02-17 13:06:35.748 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:06:25.399990+00:00'} (T+839473568 ms)
2025-02-17 13:07:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839524410 ms)
2025-02-17 13:07:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839524410 ms)
2025-02-17 13:07:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:07:25.402632+00:00', 'runtime': '839520.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.632', 'adc_vl_f': '12.28', 'adc_ic_f': '0.531', 'adc_il_f': '0.387', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.523', 'sweep_vmp': '19.656', 'sweep_pmax': '6.311', 'sweep_voc': '22.043', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839524412 ms)
2025-02-17 13:07:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:06:25.402591+00:00', 'runtime': '839460.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.635', 'adc_vl_f': '12.253', 'adc_ic_f': '0.519', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.289', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.341', 'sweep_vmp': '19.656', 'sweep_pmax': '6.311', 'sweep_voc': '22.043', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839524413 ms)
2025-02-17 13:07:36.094 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839533913 ms)
2025-02-17 13:07:36.094 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839533914 ms)
2025-02-17 13:07:36.095 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:07:25.400033+00:00'} (T+839533915 ms)
2025-02-17 13:08:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839584410 ms)
2025-02-17 13:08:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839584410 ms)
2025-02-17 13:08:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:08:25.400180+00:00', 'runtime': '839580.001', 'adc_vb_f': '12.292', 'adc_va_f': '19.482', 'adc_vl_f': '12.28', 'adc_ic_f': '0.531', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.492', 'sweep_vmp': '19.507', 'sweep_pmax': '6.674', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839584411 ms)
2025-02-17 13:08:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:07:25.402632+00:00', 'runtime': '839520.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.632', 'adc_vl_f': '12.28', 'adc_ic_f': '0.531', 'adc_il_f': '0.387', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.0', 'ahc_t': '2630.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.523', 'sweep_vmp': '19.656', 'sweep_pmax': '6.311', 'sweep_voc': '22.043', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.0', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839584412 ms)
2025-02-17 13:08:35.504 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839593324 ms)
2025-02-17 13:08:35.505 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839593324 ms)
2025-02-17 13:08:35.506 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:08:25.405142+00:00'} (T+839593325 ms)
2025-02-17 13:09:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839644408 ms)
2025-02-17 13:09:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839644409 ms)
2025-02-17 13:09:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:09:25.402832+00:00', 'runtime': '839640.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.479', 'adc_vl_f': '12.28', 'adc_ic_f': '0.524', 'adc_il_f': '0.384', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.432', 'sweep_vmp': '19.507', 'sweep_pmax': '6.674', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839644410 ms)
2025-02-17 13:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:08:25.400180+00:00', 'runtime': '839580.001', 'adc_vb_f': '12.292', 'adc_va_f': '19.482', 'adc_vl_f': '12.28', 'adc_ic_f': '0.531', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.492', 'sweep_vmp': '19.507', 'sweep_pmax': '6.674', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839644410 ms)
2025-02-17 13:09:36.171 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839653991 ms)
2025-02-17 13:09:36.172 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839653991 ms)
2025-02-17 13:09:36.173 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:09:25.400205+00:00'} (T+839653992 ms)
2025-02-17 13:10:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839704423 ms)
2025-02-17 13:10:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839704424 ms)
2025-02-17 13:10:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:10:25.402728+00:00', 'runtime': '839700.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.482', 'adc_vl_f': '12.28', 'adc_ic_f': '0.524', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.402', 'sweep_vmp': '19.507', 'sweep_pmax': '6.674', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839704428 ms)
2025-02-17 13:10:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:09:25.402832+00:00', 'runtime': '839640.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.479', 'adc_vl_f': '12.28', 'adc_ic_f': '0.524', 'adc_il_f': '0.384', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.432', 'sweep_vmp': '19.507', 'sweep_pmax': '6.674', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839704429 ms)
2025-02-17 13:10:35.876 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+839713696 ms)
2025-02-17 13:10:35.877 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+839713696 ms)
2025-02-17 13:10:35.877 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59890)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+839713697 ms)
2025-02-17 13:10:36.022 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839713842 ms)
2025-02-17 13:10:36.024 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839713843 ms)
2025-02-17 13:10:36.027 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:10:25.400132+00:00'} (T+839713846 ms)
2025-02-17 13:11:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839764409 ms)
2025-02-17 13:11:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839764410 ms)
2025-02-17 13:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:11:25.402795+00:00', 'runtime': '839760.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.632', 'adc_vl_f': '12.28', 'adc_ic_f': '0.541', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.613', 'sweep_vmp': '19.644', 'sweep_pmax': '6.764', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839764411 ms)
2025-02-17 13:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:10:25.402728+00:00', 'runtime': '839700.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.482', 'adc_vl_f': '12.28', 'adc_ic_f': '0.524', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.402', 'sweep_vmp': '19.507', 'sweep_pmax': '6.674', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839764411 ms)
2025-02-17 13:11:35.635 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839773454 ms)
2025-02-17 13:11:35.635 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839773454 ms)
2025-02-17 13:11:35.636 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:11:25.400173+00:00'} (T+839773456 ms)
2025-02-17 13:12:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839824411 ms)
2025-02-17 13:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839824412 ms)
2025-02-17 13:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:12:25.402793+00:00', 'runtime': '839820.003', 'adc_vb_f': '12.299', 'adc_va_f': '19.632', 'adc_vl_f': '12.28', 'adc_ic_f': '0.551', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.299', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.764', 'sweep_vmp': '19.644', 'sweep_pmax': '6.764', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839824413 ms)
2025-02-17 13:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:11:25.402795+00:00', 'runtime': '839760.003', 'adc_vb_f': '12.292', 'adc_va_f': '19.632', 'adc_vl_f': '12.28', 'adc_ic_f': '0.541', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.292', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.613', 'sweep_vmp': '19.644', 'sweep_pmax': '6.764', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839824414 ms)
2025-02-17 13:12:35.662 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839833481 ms)
2025-02-17 13:12:35.662 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839833482 ms)
2025-02-17 13:12:35.663 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:12:25.400153+00:00'} (T+839833483 ms)
2025-02-17 13:13:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839884411 ms)
2025-02-17 13:13:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839884412 ms)
2025-02-17 13:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:13:25.402758+00:00', 'runtime': '839880.003', 'adc_vb_f': '12.305', 'adc_va_f': '19.635', 'adc_vl_f': '12.28', 'adc_ic_f': '0.558', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.305', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.855', 'sweep_vmp': '19.644', 'sweep_pmax': '6.764', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839884413 ms)
2025-02-17 13:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:12:25.402793+00:00', 'runtime': '839820.003', 'adc_vb_f': '12.299', 'adc_va_f': '19.632', 'adc_vl_f': '12.28', 'adc_ic_f': '0.551', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.299', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.764', 'sweep_vmp': '19.644', 'sweep_pmax': '6.764', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839884414 ms)
2025-02-17 13:13:35.669 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839893488 ms)
2025-02-17 13:13:35.670 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839893489 ms)
2025-02-17 13:13:35.672 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:13:25.400135+00:00'} (T+839893491 ms)
2025-02-17 13:14:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+839944413 ms)
2025-02-17 13:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+839944413 ms)
2025-02-17 13:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:14:25.400184+00:00', 'runtime': '839940.001', 'adc_vb_f': '12.311', 'adc_va_f': '19.778', 'adc_vl_f': '12.28', 'adc_ic_f': '0.563', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.308', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.945', 'sweep_vmp': '19.785', 'sweep_pmax': '7.006', 'sweep_voc': '22.104', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839944414 ms)
2025-02-17 13:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:13:25.402758+00:00', 'runtime': '839880.003', 'adc_vb_f': '12.305', 'adc_va_f': '19.635', 'adc_vl_f': '12.28', 'adc_ic_f': '0.558', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.305', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.855', 'sweep_vmp': '19.644', 'sweep_pmax': '6.764', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+839944415 ms)
2025-02-17 13:14:36.308 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839954127 ms)
2025-02-17 13:14:36.309 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+839954128 ms)
2025-02-17 13:14:36.310 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:14:25.400128+00:00'} (T+839954129 ms)
2025-02-17 13:15:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840004410 ms)
2025-02-17 13:15:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840004410 ms)
2025-02-17 13:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:15:25.400188+00:00', 'runtime': '840000.001', 'adc_vb_f': '12.329', 'adc_va_f': '19.781', 'adc_vl_f': '12.283', 'adc_ic_f': '0.725', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.323', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.953', 'sweep_vmp': '19.785', 'sweep_pmax': '7.006', 'sweep_voc': '22.104', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840004411 ms)
2025-02-17 13:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:14:25.400184+00:00', 'runtime': '839940.001', 'adc_vb_f': '12.311', 'adc_va_f': '19.778', 'adc_vl_f': '12.28', 'adc_ic_f': '0.563', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.308', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.945', 'sweep_vmp': '19.785', 'sweep_pmax': '7.006', 'sweep_voc': '22.104', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840004412 ms)
2025-02-17 13:15:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840013429 ms)
2025-02-17 13:15:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840013430 ms)
2025-02-17 13:15:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:15:25.400002+00:00'} (T+840013431 ms)
2025-02-17 13:15:36.260 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+840014079 ms)
2025-02-17 13:15:36.260 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+840014079 ms)
2025-02-17 13:15:36.261 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840014080 ms)
2025-02-17 13:15:36.262 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+840014081 ms)
2025-02-17 13:15:36.262 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+840014081 ms)
2025-02-17 13:15:36.262 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840014082 ms)
2025-02-17 13:15:36.263 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+840014083 ms)
2025-02-17 13:15:36.264 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+840014083 ms)
2025-02-17 13:15:36.264 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840014083 ms)
2025-02-17 13:15:36.265 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+840014084 ms)
2025-02-17 13:15:36.265 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+840014084 ms)
2025-02-17 13:15:36.268 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840014087 ms)
2025-02-17 13:16:20.398 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+840058218 ms)
2025-02-17 13:16:20.399 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+840058218 ms)
2025-02-17 13:16:20.399 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60268)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840058218 ms)
2025-02-17 13:16:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840064411 ms)
2025-02-17 13:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840064411 ms)
2025-02-17 13:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:16:25.400159+00:00', 'runtime': '840060.001', 'adc_vb_f': '12.369', 'adc_va_f': '19.775', 'adc_vl_f': '12.32', 'adc_ic_f': '0.952', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.36', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '11.762', 'sweep_vmp': '19.785', 'sweep_pmax': '7.006', 'sweep_voc': '22.104', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840064413 ms)
2025-02-17 13:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:15:25.400188+00:00', 'runtime': '840000.001', 'adc_vb_f': '12.329', 'adc_va_f': '19.781', 'adc_vl_f': '12.283', 'adc_ic_f': '0.725', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.323', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.953', 'sweep_vmp': '19.785', 'sweep_pmax': '7.006', 'sweep_voc': '22.104', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840064413 ms)
2025-02-17 13:16:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840073421 ms)
2025-02-17 13:16:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840073422 ms)
2025-02-17 13:16:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:16:25.400179+00:00'} (T+840073423 ms)
2025-02-17 13:17:25.640 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840123459 ms)
2025-02-17 13:17:25.640 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840123459 ms)
2025-02-17 13:17:25.641 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:17:25.400166+00:00', 'runtime': '840120.001', 'adc_vb_f': '12.399', 'adc_va_f': '20.316', 'adc_vl_f': '12.357', 'adc_ic_f': '1.029', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.381', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.788', 'sweep_vmp': '20.325', 'sweep_pmax': '12.879', 'sweep_voc': '22.678', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840123461 ms)
2025-02-17 13:17:25.642 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:16:25.400159+00:00', 'runtime': '840060.001', 'adc_vb_f': '12.369', 'adc_va_f': '19.775', 'adc_vl_f': '12.32', 'adc_ic_f': '0.952', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.36', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.2', 'ahl_t': '1590.2', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '11.762', 'sweep_vmp': '19.785', 'sweep_pmax': '7.006', 'sweep_voc': '22.104', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840123461 ms)
2025-02-17 13:18:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840184410 ms)
2025-02-17 13:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840184411 ms)
2025-02-17 13:18:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:18:25.400891+00:00', 'runtime': '840180.001', 'adc_vb_f': '12.424', 'adc_va_f': '20.316', 'adc_vl_f': '12.393', 'adc_ic_f': '1.162', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.405', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.404', 'sweep_vmp': '20.325', 'sweep_pmax': '12.879', 'sweep_voc': '22.678', 'vb_min_daily': '12.164', 'vb_max_daily': '12.43', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840184412 ms)
2025-02-17 13:18:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:17:25.400166+00:00', 'runtime': '840120.001', 'adc_vb_f': '12.399', 'adc_va_f': '20.316', 'adc_vl_f': '12.357', 'adc_ic_f': '1.029', 'adc_il_f': '0.437', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.381', 'vb_ref': '14.414', 'ahc_r': '2630.1', 'ahc_t': '2630.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.788', 'sweep_vmp': '20.325', 'sweep_pmax': '12.879', 'sweep_voc': '22.678', 'vb_min_daily': '12.164', 'vb_max_daily': '12.427', 'ahc_daily': '0.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840184412 ms)
2025-02-17 13:19:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840244412 ms)
2025-02-17 13:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840244412 ms)
2025-02-17 13:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:19:25.400190+00:00', 'runtime': '840240.001', 'adc_vb_f': '12.457', 'adc_va_f': '20.312', 'adc_vl_f': '12.433', 'adc_ic_f': '1.336', 'adc_il_f': '0.396', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.442', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.593', 'sweep_vmp': '20.325', 'sweep_pmax': '12.879', 'sweep_voc': '22.678', 'vb_min_daily': '12.164', 'vb_max_daily': '12.46', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840244413 ms)
2025-02-17 13:19:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:18:25.400891+00:00', 'runtime': '840180.001', 'adc_vb_f': '12.424', 'adc_va_f': '20.316', 'adc_vl_f': '12.393', 'adc_ic_f': '1.162', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.405', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.404', 'sweep_vmp': '20.325', 'sweep_pmax': '12.879', 'sweep_voc': '22.678', 'vb_min_daily': '12.164', 'vb_max_daily': '12.43', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840244414 ms)
2025-02-17 13:19:35.886 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840253705 ms)
2025-02-17 13:19:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840253706 ms)
2025-02-17 13:19:35.888 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:19:25.413052+00:00'} (T+840253707 ms)
2025-02-17 13:19:46.697 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+840264516 ms)
2025-02-17 13:19:46.697 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+840264516 ms)
2025-02-17 13:19:46.698 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60488)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840264517 ms)
2025-02-17 13:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840304412 ms)
2025-02-17 13:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840304413 ms)
2025-02-17 13:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:20:25.400183+00:00', 'runtime': '840300.001', 'adc_vb_f': '12.488', 'adc_va_f': '20.502', 'adc_vl_f': '12.469', 'adc_ic_f': '1.418', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.476', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.696', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '22.949', 'vb_min_daily': '12.164', 'vb_max_daily': '12.488', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840304414 ms)
2025-02-17 13:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:19:25.400190+00:00', 'runtime': '840240.001', 'adc_vb_f': '12.457', 'adc_va_f': '20.312', 'adc_vl_f': '12.433', 'adc_ic_f': '1.336', 'adc_il_f': '0.396', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.442', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.593', 'sweep_vmp': '20.325', 'sweep_pmax': '12.879', 'sweep_voc': '22.678', 'vb_min_daily': '12.164', 'vb_max_daily': '12.46', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840304414 ms)
2025-02-17 13:20:35.711 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840313530 ms)
2025-02-17 13:20:35.712 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840313531 ms)
2025-02-17 13:20:35.713 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:20:25.405133+00:00'} (T+840313532 ms)
2025-02-17 13:21:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840364414 ms)
2025-02-17 13:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840364414 ms)
2025-02-17 13:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:21:25.402659+00:00', 'runtime': '840360.003', 'adc_vb_f': '12.521', 'adc_va_f': '20.502', 'adc_vl_f': '12.503', 'adc_ic_f': '1.43', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.877', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '22.949', 'vb_min_daily': '12.164', 'vb_max_daily': '12.521', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840364416 ms)
2025-02-17 13:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:20:25.400183+00:00', 'runtime': '840300.001', 'adc_vb_f': '12.488', 'adc_va_f': '20.502', 'adc_vl_f': '12.469', 'adc_ic_f': '1.418', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.476', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.696', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '22.949', 'vb_min_daily': '12.164', 'vb_max_daily': '12.488', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840364417 ms)
2025-02-17 13:21:35.910 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840373729 ms)
2025-02-17 13:21:35.911 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840373730 ms)
2025-02-17 13:21:35.912 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:21:25.400030+00:00'} (T+840373731 ms)
2025-02-17 13:21:55.946 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+840393765 ms)
2025-02-17 13:21:55.946 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+840393766 ms)
2025-02-17 13:21:55.947 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60628)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840393766 ms)
2025-02-17 13:22:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840424415 ms)
2025-02-17 13:22:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840424416 ms)
2025-02-17 13:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:22:25.402831+00:00', 'runtime': '840420.003', 'adc_vb_f': '12.558', 'adc_va_f': '20.505', 'adc_vl_f': '12.509', 'adc_ic_f': '1.561', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.553', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '22.949', 'vb_min_daily': '12.164', 'vb_max_daily': '12.558', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840424417 ms)
2025-02-17 13:22:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:21:25.402659+00:00', 'runtime': '840360.003', 'adc_vb_f': '12.521', 'adc_va_f': '20.502', 'adc_vl_f': '12.503', 'adc_ic_f': '1.43', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.414', 'ahc_r': '2630.2', 'ahc_t': '2630.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.877', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '22.949', 'vb_min_daily': '12.164', 'vb_max_daily': '12.521', 'ahc_daily': '0.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840424418 ms)
2025-02-17 13:22:35.712 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840433531 ms)
2025-02-17 13:22:35.713 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840433532 ms)
2025-02-17 13:22:35.714 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:22:25.400230+00:00'} (T+840433533 ms)
2025-02-17 13:23:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840484414 ms)
2025-02-17 13:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840484415 ms)
2025-02-17 13:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:23:25.402785+00:00', 'runtime': '840480.003', 'adc_vb_f': '12.555', 'adc_va_f': '20.505', 'adc_vl_f': '12.537', 'adc_ic_f': '1.539', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.356', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.585', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840484419 ms)
2025-02-17 13:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:22:25.402831+00:00', 'runtime': '840420.003', 'adc_vb_f': '12.558', 'adc_va_f': '20.505', 'adc_vl_f': '12.509', 'adc_ic_f': '1.561', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.553', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '22.949', 'vb_min_daily': '12.164', 'vb_max_daily': '12.558', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840484419 ms)
2025-02-17 13:23:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840493439 ms)
2025-02-17 13:23:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840493440 ms)
2025-02-17 13:23:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:23:25.400168+00:00'} (T+840493441 ms)
2025-02-17 13:24:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840544412 ms)
2025-02-17 13:24:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840544412 ms)
2025-02-17 13:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:24:25.400247+00:00', 'runtime': '840540.001', 'adc_vb_f': '12.598', 'adc_va_f': '20.502', 'adc_vl_f': '12.585', 'adc_ic_f': '1.635', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '20.594', 'sweep_vmp': '20.551', 'sweep_pmax': '19.522', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.598', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840544416 ms)
2025-02-17 13:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:23:25.402785+00:00', 'runtime': '840480.003', 'adc_vb_f': '12.555', 'adc_va_f': '20.505', 'adc_vl_f': '12.537', 'adc_ic_f': '1.539', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.356', 'sweep_vmp': '20.547', 'sweep_pmax': '17.877', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.585', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840544416 ms)
2025-02-17 13:24:35.656 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840553476 ms)
2025-02-17 13:24:35.657 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840553476 ms)
2025-02-17 13:24:35.662 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:24:25.400245+00:00'} (T+840553481 ms)
2025-02-17 13:25:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840604418 ms)
2025-02-17 13:25:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840604419 ms)
2025-02-17 13:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:25:25.400200+00:00', 'runtime': '840600.001', 'adc_vb_f': '12.634', 'adc_va_f': '20.502', 'adc_vl_f': '12.585', 'adc_ic_f': '1.655', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '20.896', 'sweep_vmp': '20.551', 'sweep_pmax': '19.522', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.634', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840604420 ms)
2025-02-17 13:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:24:25.400247+00:00', 'runtime': '840540.001', 'adc_vb_f': '12.598', 'adc_va_f': '20.502', 'adc_vl_f': '12.585', 'adc_ic_f': '1.635', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '20.594', 'sweep_vmp': '20.551', 'sweep_pmax': '19.522', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.598', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840604420 ms)
2025-02-17 13:25:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840613435 ms)
2025-02-17 13:25:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840613435 ms)
2025-02-17 13:25:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:25:25.400147+00:00'} (T+840613443 ms)
2025-02-17 13:26:09.338 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+840647157 ms)
2025-02-17 13:26:09.338 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+840647157 ms)
2025-02-17 13:26:09.339 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60908)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840647158 ms)
2025-02-17 13:26:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840664413 ms)
2025-02-17 13:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840664414 ms)
2025-02-17 13:26:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:26:25.400178+00:00', 'runtime': '840660.001', 'adc_vb_f': '12.656', 'adc_va_f': '20.502', 'adc_vl_f': '12.622', 'adc_ic_f': '1.715', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '14.414', 'ahc_r': '2630.4', 'ahc_t': '2630.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '21.697', 'sweep_vmp': '20.551', 'sweep_pmax': '19.522', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.662', 'ahc_daily': '0.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840664415 ms)
2025-02-17 13:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:25:25.400200+00:00', 'runtime': '840600.001', 'adc_vb_f': '12.634', 'adc_va_f': '20.502', 'adc_vl_f': '12.585', 'adc_ic_f': '1.655', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '14.414', 'ahc_r': '2630.3', 'ahc_t': '2630.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '20.896', 'sweep_vmp': '20.551', 'sweep_pmax': '19.522', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.634', 'ahc_daily': '0.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840664415 ms)
2025-02-17 13:27:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840724412 ms)
2025-02-17 13:27:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840724412 ms)
2025-02-17 13:27:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:27:25.400164+00:00', 'runtime': '840720.001', 'adc_vb_f': '12.674', 'adc_va_f': '20.554', 'adc_vl_f': '12.653', 'adc_ic_f': '1.744', 'adc_il_f': '0.372', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '14.414', 'ahc_r': '2630.4', 'ahc_t': '2630.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '22.104', 'sweep_vmp': '20.587', 'sweep_pmax': '21.908', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.674', 'ahc_daily': '0.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840724413 ms)
2025-02-17 13:27:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:26:25.400178+00:00', 'runtime': '840660.001', 'adc_vb_f': '12.656', 'adc_va_f': '20.502', 'adc_vl_f': '12.622', 'adc_ic_f': '1.715', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '14.414', 'ahc_r': '2630.4', 'ahc_t': '2630.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '21.697', 'sweep_vmp': '20.551', 'sweep_pmax': '19.522', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.662', 'ahc_daily': '0.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840724414 ms)
2025-02-17 13:28:05.481 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+840763301 ms)
2025-02-17 13:28:05.482 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+840763301 ms)
2025-02-17 13:28:05.482 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840763302 ms)
2025-02-17 13:28:05.484 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+840763303 ms)
2025-02-17 13:28:05.484 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+840763303 ms)
2025-02-17 13:28:05.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+840763304 ms)
2025-02-17 13:28:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840784412 ms)
2025-02-17 13:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840784412 ms)
2025-02-17 13:28:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:28:25.400257+00:00', 'runtime': '840780.001', 'adc_vb_f': '12.704', 'adc_va_f': '20.554', 'adc_vl_f': '12.662', 'adc_ic_f': '1.749', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '14.414', 'ahc_r': '2630.4', 'ahc_t': '2630.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '22.165', 'sweep_vmp': '20.587', 'sweep_pmax': '21.908', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.704', 'ahc_daily': '0.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840784413 ms)
2025-02-17 13:28:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:27:25.400164+00:00', 'runtime': '840720.001', 'adc_vb_f': '12.674', 'adc_va_f': '20.554', 'adc_vl_f': '12.653', 'adc_ic_f': '1.744', 'adc_il_f': '0.372', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '14.414', 'ahc_r': '2630.4', 'ahc_t': '2630.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '22.104', 'sweep_vmp': '20.587', 'sweep_pmax': '21.908', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.674', 'ahc_daily': '0.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840784414 ms)
2025-02-17 13:28:36.095 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840793914 ms)
2025-02-17 13:28:36.096 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840793915 ms)
2025-02-17 13:28:36.097 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:28:25.400345+00:00'} (T+840793917 ms)
2025-02-17 13:29:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840844423 ms)
2025-02-17 13:29:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840844424 ms)
2025-02-17 13:29:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:29:25.400541+00:00', 'runtime': '840840.001', 'adc_vb_f': '12.714', 'adc_va_f': '20.554', 'adc_vl_f': '12.695', 'adc_ic_f': '1.834', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '14.414', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.327', 'sweep_vmp': '20.587', 'sweep_pmax': '21.908', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.714', 'ahc_daily': '0.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840844425 ms)
2025-02-17 13:29:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:28:25.400257+00:00', 'runtime': '840780.001', 'adc_vb_f': '12.704', 'adc_va_f': '20.554', 'adc_vl_f': '12.662', 'adc_ic_f': '1.749', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '14.414', 'ahc_r': '2630.4', 'ahc_t': '2630.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '22.165', 'sweep_vmp': '20.587', 'sweep_pmax': '21.908', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.704', 'ahc_daily': '0.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840844425 ms)
2025-02-17 13:29:35.894 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840853713 ms)
2025-02-17 13:29:35.894 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+840853713 ms)
2025-02-17 13:29:35.899 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:29:25.400198+00:00'} (T+840853718 ms)
2025-02-17 13:30:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840904408 ms)
2025-02-17 13:30:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840904409 ms)
2025-02-17 13:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:30:25.400164+00:00', 'runtime': '840900.001', 'adc_vb_f': '12.75', 'adc_va_f': '20.654', 'adc_vl_f': '12.704', 'adc_ic_f': '1.889', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.738', 'vb_ref': '14.414', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.082', 'sweep_vmp': '20.685', 'sweep_pmax': '23.659', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.75', 'ahc_daily': '0.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840904410 ms)
2025-02-17 13:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:29:25.400541+00:00', 'runtime': '840840.001', 'adc_vb_f': '12.714', 'adc_va_f': '20.554', 'adc_vl_f': '12.695', 'adc_ic_f': '1.834', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '14.414', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.327', 'sweep_vmp': '20.587', 'sweep_pmax': '21.908', 'sweep_voc': '23.059', 'vb_min_daily': '12.164', 'vb_max_daily': '12.714', 'ahc_daily': '0.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840904410 ms)
2025-02-17 13:31:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+840964409 ms)
2025-02-17 13:31:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+840964409 ms)
2025-02-17 13:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:31:25.400202+00:00', 'runtime': '840960.001', 'adc_vb_f': '12.753', 'adc_va_f': '20.654', 'adc_vl_f': '12.738', 'adc_ic_f': '1.906', 'adc_il_f': '0.387', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '14.414', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.339', 'sweep_vmp': '20.685', 'sweep_pmax': '23.659', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.756', 'ahc_daily': '0.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840964410 ms)
2025-02-17 13:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:30:25.400164+00:00', 'runtime': '840900.001', 'adc_vb_f': '12.75', 'adc_va_f': '20.654', 'adc_vl_f': '12.704', 'adc_ic_f': '1.889', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.738', 'vb_ref': '14.414', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.3', 'ahl_t': '1590.3', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.082', 'sweep_vmp': '20.685', 'sweep_pmax': '23.659', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.75', 'ahc_daily': '0.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+840964411 ms)
2025-02-17 13:32:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841024411 ms)
2025-02-17 13:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841024411 ms)
2025-02-17 13:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:32:25.400246+00:00', 'runtime': '841020.001', 'adc_vb_f': '12.784', 'adc_va_f': '20.654', 'adc_vl_f': '12.738', 'adc_ic_f': '1.911', 'adc_il_f': '0.391', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.772', 'vb_ref': '14.384', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.429', 'sweep_vmp': '20.685', 'sweep_pmax': '23.659', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.787', 'ahc_daily': '0.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841024412 ms)
2025-02-17 13:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:31:25.400202+00:00', 'runtime': '840960.001', 'adc_vb_f': '12.753', 'adc_va_f': '20.654', 'adc_vl_f': '12.738', 'adc_ic_f': '1.906', 'adc_il_f': '0.387', 't_hs': '5', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '14.414', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.339', 'sweep_vmp': '20.685', 'sweep_pmax': '23.659', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.756', 'ahc_daily': '0.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841024413 ms)
2025-02-17 13:32:37.908 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841035727 ms)
2025-02-17 13:32:37.908 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841035727 ms)
2025-02-17 13:32:37.909 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33086)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841035728 ms)
2025-02-17 13:33:25.691 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841083510 ms)
2025-02-17 13:33:25.692 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841083511 ms)
2025-02-17 13:33:25.693 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:33:25.400153+00:00', 'runtime': '841080.001', 'adc_vb_f': '12.787', 'adc_va_f': '20.599', 'adc_vl_f': '12.738', 'adc_ic_f': '1.916', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.784', 'vb_ref': '14.384', 'ahc_r': '2630.6', 'ahc_t': '2630.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.429', 'sweep_vmp': '20.633', 'sweep_pmax': '24.641', 'sweep_voc': '23.077', 'vb_min_daily': '12.164', 'vb_max_daily': '12.787', 'ahc_daily': '0.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841083512 ms)
2025-02-17 13:33:25.693 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:32:25.400246+00:00', 'runtime': '841020.001', 'adc_vb_f': '12.784', 'adc_va_f': '20.654', 'adc_vl_f': '12.738', 'adc_ic_f': '1.911', 'adc_il_f': '0.391', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.772', 'vb_ref': '14.384', 'ahc_r': '2630.5', 'ahc_t': '2630.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.429', 'sweep_vmp': '20.685', 'sweep_pmax': '23.659', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.787', 'ahc_daily': '0.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841083512 ms)
2025-02-17 13:33:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841093420 ms)
2025-02-17 13:33:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841093421 ms)
2025-02-17 13:33:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:33:25.400308+00:00'} (T+841093422 ms)
2025-02-17 13:34:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841144414 ms)
2025-02-17 13:34:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841144415 ms)
2025-02-17 13:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:34:25.400061+00:00', 'runtime': '841140.001', 'adc_vb_f': '12.79', 'adc_va_f': '20.599', 'adc_vl_f': '12.772', 'adc_ic_f': '1.95', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.384', 'ahc_r': '2630.6', 'ahc_t': '2630.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.928', 'sweep_vmp': '20.633', 'sweep_pmax': '24.641', 'sweep_voc': '23.077', 'vb_min_daily': '12.164', 'vb_max_daily': '12.799', 'ahc_daily': '0.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841144416 ms)
2025-02-17 13:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:33:25.400153+00:00', 'runtime': '841080.001', 'adc_vb_f': '12.787', 'adc_va_f': '20.599', 'adc_vl_f': '12.738', 'adc_ic_f': '1.916', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.784', 'vb_ref': '14.384', 'ahc_r': '2630.6', 'ahc_t': '2630.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.429', 'sweep_vmp': '20.633', 'sweep_pmax': '24.641', 'sweep_voc': '23.077', 'vb_min_daily': '12.164', 'vb_max_daily': '12.787', 'ahc_daily': '0.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841144417 ms)
2025-02-17 13:34:35.820 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841153639 ms)
2025-02-17 13:34:35.820 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841153639 ms)
2025-02-17 13:34:35.825 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:34:25.400189+00:00'} (T+841153644 ms)
2025-02-17 13:35:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841204414 ms)
2025-02-17 13:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841204414 ms)
2025-02-17 13:35:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:35:25.400192+00:00', 'runtime': '841200.001', 'adc_vb_f': '12.808', 'adc_va_f': '20.599', 'adc_vl_f': '12.775', 'adc_ic_f': '1.911', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.808', 'vb_ref': '14.384', 'ahc_r': '2630.6', 'ahc_t': '2630.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.429', 'sweep_vmp': '20.633', 'sweep_pmax': '24.641', 'sweep_voc': '23.077', 'vb_min_daily': '12.164', 'vb_max_daily': '12.817', 'ahc_daily': '0.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841204418 ms)
2025-02-17 13:35:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:34:25.400061+00:00', 'runtime': '841140.001', 'adc_vb_f': '12.79', 'adc_va_f': '20.599', 'adc_vl_f': '12.772', 'adc_ic_f': '1.95', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.384', 'ahc_r': '2630.6', 'ahc_t': '2630.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.928', 'sweep_vmp': '20.633', 'sweep_pmax': '24.641', 'sweep_voc': '23.077', 'vb_min_daily': '12.164', 'vb_max_daily': '12.799', 'ahc_daily': '0.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841204419 ms)
2025-02-17 13:35:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841213435 ms)
2025-02-17 13:35:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841213436 ms)
2025-02-17 13:35:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:35:25.400025+00:00'} (T+841213437 ms)
2025-02-17 13:36:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841264416 ms)
2025-02-17 13:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841264416 ms)
2025-02-17 13:36:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:36:25.400211+00:00', 'runtime': '841260.001', 'adc_vb_f': '12.814', 'adc_va_f': '20.554', 'adc_vl_f': '12.775', 'adc_ic_f': '1.894', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.811', 'vb_ref': '14.384', 'ahc_r': '2630.7', 'ahc_t': '2630.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.263', 'sweep_vmp': '20.584', 'sweep_pmax': '24.414', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.82', 'ahc_daily': '0.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841264420 ms)
2025-02-17 13:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:35:25.400192+00:00', 'runtime': '841200.001', 'adc_vb_f': '12.808', 'adc_va_f': '20.599', 'adc_vl_f': '12.775', 'adc_ic_f': '1.911', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.808', 'vb_ref': '14.384', 'ahc_r': '2630.6', 'ahc_t': '2630.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.429', 'sweep_vmp': '20.633', 'sweep_pmax': '24.641', 'sweep_voc': '23.077', 'vb_min_daily': '12.164', 'vb_max_daily': '12.817', 'ahc_daily': '0.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841264420 ms)
2025-02-17 13:36:36.383 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841274202 ms)
2025-02-17 13:36:36.384 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841274203 ms)
2025-02-17 13:36:36.388 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:36:25.400224+00:00'} (T+841274208 ms)
2025-02-17 13:36:52.297 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841290116 ms)
2025-02-17 13:36:52.297 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841290117 ms)
2025-02-17 13:36:52.298 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33368)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841290117 ms)
2025-02-17 13:37:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841324416 ms)
2025-02-17 13:37:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841324416 ms)
2025-02-17 13:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:37:25.400159+00:00', 'runtime': '841320.001', 'adc_vb_f': '12.823', 'adc_va_f': '20.551', 'adc_vl_f': '12.787', 'adc_ic_f': '1.942', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.82', 'vb_ref': '14.384', 'ahc_r': '2630.7', 'ahc_t': '2630.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.882', 'sweep_vmp': '20.584', 'sweep_pmax': '24.414', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.823', 'ahc_daily': '0.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841324417 ms)
2025-02-17 13:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:36:25.400211+00:00', 'runtime': '841260.001', 'adc_vb_f': '12.814', 'adc_va_f': '20.554', 'adc_vl_f': '12.775', 'adc_ic_f': '1.894', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.811', 'vb_ref': '14.384', 'ahc_r': '2630.7', 'ahc_t': '2630.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.263', 'sweep_vmp': '20.584', 'sweep_pmax': '24.414', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.82', 'ahc_daily': '0.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841324418 ms)
2025-02-17 13:37:35.648 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841333468 ms)
2025-02-17 13:37:35.649 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841333468 ms)
2025-02-17 13:37:35.651 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:37:25.400413+00:00'} (T+841333470 ms)
2025-02-17 13:37:55.460 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841353279 ms)
2025-02-17 13:37:55.460 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841353279 ms)
2025-02-17 13:37:55.461 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33436)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841353280 ms)
2025-02-17 13:38:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841384416 ms)
2025-02-17 13:38:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841384417 ms)
2025-02-17 13:38:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:38:25.402721+00:00', 'runtime': '841380.003', 'adc_vb_f': '12.833', 'adc_va_f': '20.554', 'adc_vl_f': '12.814', 'adc_ic_f': '2.034', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.384', 'ahc_r': '2630.7', 'ahc_t': '2630.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '26.151', 'sweep_vmp': '20.584', 'sweep_pmax': '24.414', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.836', 'ahc_daily': '0.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841384418 ms)
2025-02-17 13:38:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:37:25.400159+00:00', 'runtime': '841320.001', 'adc_vb_f': '12.823', 'adc_va_f': '20.551', 'adc_vl_f': '12.787', 'adc_ic_f': '1.942', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.82', 'vb_ref': '14.384', 'ahc_r': '2630.7', 'ahc_t': '2630.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.882', 'sweep_vmp': '20.584', 'sweep_pmax': '24.414', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.823', 'ahc_daily': '0.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841384418 ms)
2025-02-17 13:38:35.752 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841393571 ms)
2025-02-17 13:38:35.753 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841393572 ms)
2025-02-17 13:38:35.757 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:38:25.400127+00:00'} (T+841393576 ms)
2025-02-17 13:39:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841444420 ms)
2025-02-17 13:39:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841444420 ms)
2025-02-17 13:39:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:39:25.402703+00:00', 'runtime': '841440.003', 'adc_vb_f': '12.833', 'adc_va_f': '20.554', 'adc_vl_f': '12.814', 'adc_ic_f': '1.937', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.83', 'vb_ref': '14.384', 'ahc_r': '2630.8', 'ahc_t': '2630.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.807', 'sweep_vmp': '20.584', 'sweep_pmax': '26.196', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.836', 'ahc_daily': '0.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841444421 ms)
2025-02-17 13:39:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:38:25.402721+00:00', 'runtime': '841380.003', 'adc_vb_f': '12.833', 'adc_va_f': '20.554', 'adc_vl_f': '12.814', 'adc_ic_f': '2.034', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.384', 'ahc_r': '2630.7', 'ahc_t': '2630.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '26.151', 'sweep_vmp': '20.584', 'sweep_pmax': '24.414', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.836', 'ahc_daily': '0.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841444422 ms)
2025-02-17 13:40:25.691 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841503511 ms)
2025-02-17 13:40:25.692 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841503511 ms)
2025-02-17 13:40:25.692 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33596)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841503511 ms)
2025-02-17 13:40:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841504406 ms)
2025-02-17 13:40:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841504406 ms)
2025-02-17 13:40:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:40:25.400012+00:00', 'runtime': '841500.001', 'adc_vb_f': '12.823', 'adc_va_f': '20.551', 'adc_vl_f': '12.814', 'adc_ic_f': '1.848', 'adc_il_f': '0.389', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.827', 'vb_ref': '14.384', 'ahc_r': '2630.8', 'ahc_t': '2630.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.659', 'sweep_vmp': '20.584', 'sweep_pmax': '26.196', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.836', 'ahc_daily': '0.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841504407 ms)
2025-02-17 13:40:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:39:25.402703+00:00', 'runtime': '841440.003', 'adc_vb_f': '12.833', 'adc_va_f': '20.554', 'adc_vl_f': '12.814', 'adc_ic_f': '1.937', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.83', 'vb_ref': '14.384', 'ahc_r': '2630.8', 'ahc_t': '2630.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '24.807', 'sweep_vmp': '20.584', 'sweep_pmax': '26.196', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.836', 'ahc_daily': '0.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841504407 ms)
2025-02-17 13:41:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841564423 ms)
2025-02-17 13:41:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841564424 ms)
2025-02-17 13:41:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:41:25.400185+00:00', 'runtime': '841560.001', 'adc_vb_f': '12.866', 'adc_va_f': '20.554', 'adc_vl_f': '12.848', 'adc_ic_f': '2.206', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.854', 'vb_ref': '14.384', 'ahc_r': '2630.8', 'ahc_t': '2630.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '28.37', 'sweep_vmp': '20.584', 'sweep_pmax': '26.196', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.863', 'ahc_daily': '0.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841564425 ms)
2025-02-17 13:41:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:40:25.400012+00:00', 'runtime': '841500.001', 'adc_vb_f': '12.823', 'adc_va_f': '20.551', 'adc_vl_f': '12.814', 'adc_ic_f': '1.848', 'adc_il_f': '0.389', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.827', 'vb_ref': '14.384', 'ahc_r': '2630.8', 'ahc_t': '2630.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.659', 'sweep_vmp': '20.584', 'sweep_pmax': '26.196', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.836', 'ahc_daily': '0.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841564425 ms)
2025-02-17 13:41:35.740 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841573560 ms)
2025-02-17 13:41:35.741 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841573560 ms)
2025-02-17 13:41:35.743 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:41:25.413054+00:00'} (T+841573562 ms)
2025-02-17 13:42:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841624424 ms)
2025-02-17 13:42:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841624424 ms)
2025-02-17 13:42:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:42:25.400081+00:00', 'runtime': '841620.001', 'adc_vb_f': '12.897', 'adc_va_f': '20.703', 'adc_vl_f': '12.851', 'adc_ic_f': '2.21', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.885', 'vb_ref': '14.384', 'ahc_r': '2630.9', 'ahc_t': '2630.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '28.461', 'sweep_vmp': '20.709', 'sweep_pmax': '28.899', 'sweep_voc': '23.157', 'vb_min_daily': '12.164', 'vb_max_daily': '12.9', 'ahc_daily': '0.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841624425 ms)
2025-02-17 13:42:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:41:25.400185+00:00', 'runtime': '841560.001', 'adc_vb_f': '12.866', 'adc_va_f': '20.554', 'adc_vl_f': '12.848', 'adc_ic_f': '2.206', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.854', 'vb_ref': '14.384', 'ahc_r': '2630.8', 'ahc_t': '2630.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '28.37', 'sweep_vmp': '20.584', 'sweep_pmax': '26.196', 'sweep_voc': '23.068', 'vb_min_daily': '12.164', 'vb_max_daily': '12.863', 'ahc_daily': '0.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841624425 ms)
2025-02-17 13:42:35.666 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841633485 ms)
2025-02-17 13:42:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841633485 ms)
2025-02-17 13:42:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:42:25.409122+00:00'} (T+841633486 ms)
2025-02-17 13:43:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841684408 ms)
2025-02-17 13:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841684409 ms)
2025-02-17 13:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:43:25.402709+00:00', 'runtime': '841680.003', 'adc_vb_f': '12.903', 'adc_va_f': '20.703', 'adc_vl_f': '12.891', 'adc_ic_f': '2.36', 'adc_il_f': '0.399', 't_hs': '6', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.384', 'ahc_r': '2630.9', 'ahc_t': '2630.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '30.439', 'sweep_vmp': '20.709', 'sweep_pmax': '28.899', 'sweep_voc': '23.157', 'vb_min_daily': '12.164', 'vb_max_daily': '12.903', 'ahc_daily': '0.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841684411 ms)
2025-02-17 13:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:42:25.400081+00:00', 'runtime': '841620.001', 'adc_vb_f': '12.897', 'adc_va_f': '20.703', 'adc_vl_f': '12.851', 'adc_ic_f': '2.21', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.885', 'vb_ref': '14.384', 'ahc_r': '2630.9', 'ahc_t': '2630.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '28.461', 'sweep_vmp': '20.709', 'sweep_pmax': '28.899', 'sweep_voc': '23.157', 'vb_min_daily': '12.164', 'vb_max_daily': '12.9', 'ahc_daily': '0.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841684411 ms)
2025-02-17 13:43:35.753 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841693572 ms)
2025-02-17 13:43:35.753 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841693572 ms)
2025-02-17 13:43:35.754 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:43:25.400132+00:00'} (T+841693573 ms)
2025-02-17 13:44:09.053 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841726872 ms)
2025-02-17 13:44:09.053 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841726872 ms)
2025-02-17 13:44:09.053 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33834)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841726873 ms)
2025-02-17 13:44:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841744412 ms)
2025-02-17 13:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841744413 ms)
2025-02-17 13:44:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:44:25.402630+00:00', 'runtime': '841740.003', 'adc_vb_f': '12.939', 'adc_va_f': '20.706', 'adc_vl_f': '12.915', 'adc_ic_f': '2.505', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.93', 'vb_ref': '14.355', 'ahc_r': '2631.0', 'ahc_t': '2631.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '32.432', 'sweep_vmp': '20.709', 'sweep_pmax': '28.899', 'sweep_voc': '23.157', 'vb_min_daily': '12.164', 'vb_max_daily': '12.939', 'ahc_daily': '1.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841744416 ms)
2025-02-17 13:44:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:43:25.402709+00:00', 'runtime': '841680.003', 'adc_vb_f': '12.903', 'adc_va_f': '20.703', 'adc_vl_f': '12.891', 'adc_ic_f': '2.36', 'adc_il_f': '0.399', 't_hs': '6', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.384', 'ahc_r': '2630.9', 'ahc_t': '2630.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '30.439', 'sweep_vmp': '20.709', 'sweep_pmax': '28.899', 'sweep_voc': '23.157', 'vb_min_daily': '12.164', 'vb_max_daily': '12.903', 'ahc_daily': '0.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841744417 ms)
2025-02-17 13:44:36.074 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841753893 ms)
2025-02-17 13:44:36.075 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841753894 ms)
2025-02-17 13:44:36.076 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:44:25.400034+00:00'} (T+841753895 ms)
2025-02-17 13:44:38.183 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841756002 ms)
2025-02-17 13:44:38.183 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841756003 ms)
2025-02-17 13:44:38.184 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33866)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841756003 ms)
2025-02-17 13:45:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841804414 ms)
2025-02-17 13:45:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841804414 ms)
2025-02-17 13:45:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:45:25.400198+00:00', 'runtime': '841800.001', 'adc_vb_f': '12.973', 'adc_va_f': '20.755', 'adc_vl_f': '12.927', 'adc_ic_f': '2.585', 'adc_il_f': '0.43', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.958', 'vb_ref': '14.355', 'ahc_r': '2631.0', 'ahc_t': '2631.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '33.489', 'sweep_vmp': '20.795', 'sweep_pmax': '32.794', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '12.976', 'ahc_daily': '1.0', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841804418 ms)
2025-02-17 13:45:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:44:25.402630+00:00', 'runtime': '841740.003', 'adc_vb_f': '12.939', 'adc_va_f': '20.706', 'adc_vl_f': '12.915', 'adc_ic_f': '2.505', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.93', 'vb_ref': '14.355', 'ahc_r': '2631.0', 'ahc_t': '2631.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.4', 'ahl_t': '1590.4', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '32.432', 'sweep_vmp': '20.709', 'sweep_pmax': '28.899', 'sweep_voc': '23.157', 'vb_min_daily': '12.164', 'vb_max_daily': '12.939', 'ahc_daily': '1.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841804419 ms)
2025-02-17 13:46:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841864413 ms)
2025-02-17 13:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841864414 ms)
2025-02-17 13:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:46:25.400207+00:00', 'runtime': '841860.001', 'adc_vb_f': '12.976', 'adc_va_f': '20.758', 'adc_vl_f': '12.964', 'adc_ic_f': '2.662', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.973', 'vb_ref': '14.355', 'ahc_r': '2631.0', 'ahc_t': '2631.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.47', 'sweep_vmp': '20.795', 'sweep_pmax': '32.794', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '12.976', 'ahc_daily': '1.0', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841864417 ms)
2025-02-17 13:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:45:25.400198+00:00', 'runtime': '841800.001', 'adc_vb_f': '12.973', 'adc_va_f': '20.755', 'adc_vl_f': '12.927', 'adc_ic_f': '2.585', 'adc_il_f': '0.43', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.958', 'vb_ref': '14.355', 'ahc_r': '2631.0', 'ahc_t': '2631.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '33.489', 'sweep_vmp': '20.795', 'sweep_pmax': '32.794', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '12.976', 'ahc_daily': '1.0', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841864418 ms)
2025-02-17 13:46:52.404 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841890223 ms)
2025-02-17 13:46:52.404 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841890223 ms)
2025-02-17 13:46:52.405 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34018)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841890224 ms)
2025-02-17 13:47:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841924418 ms)
2025-02-17 13:47:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841924418 ms)
2025-02-17 13:47:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:47:25.400186+00:00', 'runtime': '841920.001', 'adc_vb_f': '13.01', 'adc_va_f': '20.755', 'adc_vl_f': '12.967', 'adc_ic_f': '2.694', 'adc_il_f': '0.374', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.991', 'vb_ref': '14.355', 'ahc_r': '2631.1', 'ahc_t': '2631.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.998', 'sweep_vmp': '20.795', 'sweep_pmax': '32.794', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.01', 'ahc_daily': '1.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841924420 ms)
2025-02-17 13:47:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:46:25.400207+00:00', 'runtime': '841860.001', 'adc_vb_f': '12.976', 'adc_va_f': '20.758', 'adc_vl_f': '12.964', 'adc_ic_f': '2.662', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.973', 'vb_ref': '14.355', 'ahc_r': '2631.0', 'ahc_t': '2631.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.47', 'sweep_vmp': '20.795', 'sweep_pmax': '32.794', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '12.976', 'ahc_daily': '1.0', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841924420 ms)
2025-02-17 13:47:39.562 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+841937381 ms)
2025-02-17 13:47:39.562 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+841937382 ms)
2025-02-17 13:47:39.563 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34068)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+841937382 ms)
2025-02-17 13:48:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+841984418 ms)
2025-02-17 13:48:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+841984419 ms)
2025-02-17 13:48:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:48:25.400147+00:00', 'runtime': '841980.001', 'adc_vb_f': '13.016', 'adc_va_f': '20.755', 'adc_vl_f': '12.97', 'adc_ic_f': '2.698', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.013', 'vb_ref': '14.355', 'ahc_r': '2631.1', 'ahc_t': '2631.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.059', 'sweep_vmp': '20.786', 'sweep_pmax': '35.27', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.016', 'ahc_daily': '1.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841984423 ms)
2025-02-17 13:48:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:47:25.400186+00:00', 'runtime': '841920.001', 'adc_vb_f': '13.01', 'adc_va_f': '20.755', 'adc_vl_f': '12.967', 'adc_ic_f': '2.694', 'adc_il_f': '0.374', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.991', 'vb_ref': '14.355', 'ahc_r': '2631.1', 'ahc_t': '2631.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.998', 'sweep_vmp': '20.795', 'sweep_pmax': '32.794', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.01', 'ahc_daily': '1.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+841984423 ms)
2025-02-17 13:48:35.627 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841993446 ms)
2025-02-17 13:48:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+841993447 ms)
2025-02-17 13:48:35.629 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:48:25.413049+00:00'} (T+841993448 ms)
2025-02-17 13:48:51.714 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842009533 ms)
2025-02-17 13:48:51.714 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842009534 ms)
2025-02-17 13:48:51.715 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34148)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842009534 ms)
2025-02-17 13:49:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842044420 ms)
2025-02-17 13:49:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842044421 ms)
2025-02-17 13:49:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:49:25.402761+00:00', 'runtime': '842040.003', 'adc_vb_f': '13.016', 'adc_va_f': '20.755', 'adc_vl_f': '12.979', 'adc_ic_f': '2.645', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.016', 'vb_ref': '14.355', 'ahc_r': '2631.2', 'ahc_t': '2631.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.425', 'sweep_vmp': '20.786', 'sweep_pmax': '35.27', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.016', 'ahc_daily': '1.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842044425 ms)
2025-02-17 13:49:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:48:25.400147+00:00', 'runtime': '841980.001', 'adc_vb_f': '13.016', 'adc_va_f': '20.755', 'adc_vl_f': '12.97', 'adc_ic_f': '2.698', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.013', 'vb_ref': '14.355', 'ahc_r': '2631.1', 'ahc_t': '2631.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.059', 'sweep_vmp': '20.786', 'sweep_pmax': '35.27', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.016', 'ahc_daily': '1.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842044425 ms)
2025-02-17 13:50:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842104406 ms)
2025-02-17 13:50:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842104406 ms)
2025-02-17 13:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:50:25.400563+00:00', 'runtime': '842100.001', 'adc_vb_f': '13.019', 'adc_va_f': '20.755', 'adc_vl_f': '13.004', 'adc_ic_f': '2.732', 'adc_il_f': '0.379', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.016', 'vb_ref': '14.355', 'ahc_r': '2631.2', 'ahc_t': '2631.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.557', 'sweep_vmp': '20.786', 'sweep_pmax': '35.27', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.019', 'ahc_daily': '1.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842104410 ms)
2025-02-17 13:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:49:25.402761+00:00', 'runtime': '842040.003', 'adc_vb_f': '13.016', 'adc_va_f': '20.755', 'adc_vl_f': '12.979', 'adc_ic_f': '2.645', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.016', 'vb_ref': '14.355', 'ahc_r': '2631.2', 'ahc_t': '2631.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.425', 'sweep_vmp': '20.786', 'sweep_pmax': '35.27', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.016', 'ahc_daily': '1.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842104411 ms)
2025-02-17 13:51:26.606 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842164425 ms)
2025-02-17 13:51:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842164425 ms)
2025-02-17 13:51:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:51:25.400192+00:00', 'runtime': '842160.001', 'adc_vb_f': '13.049', 'adc_va_f': '20.85', 'adc_vl_f': '13.004', 'adc_ic_f': '2.793', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.04', 'vb_ref': '14.355', 'ahc_r': '2631.3', 'ahc_t': '2631.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.433', 'sweep_vmp': '20.892', 'sweep_pmax': '36.327', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.052', 'ahc_daily': '1.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842164430 ms)
2025-02-17 13:51:26.611 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:50:25.400563+00:00', 'runtime': '842100.001', 'adc_vb_f': '13.019', 'adc_va_f': '20.755', 'adc_vl_f': '13.004', 'adc_ic_f': '2.732', 'adc_il_f': '0.379', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.016', 'vb_ref': '14.355', 'ahc_r': '2631.2', 'ahc_t': '2631.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.557', 'sweep_vmp': '20.786', 'sweep_pmax': '35.27', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.019', 'ahc_daily': '1.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842164430 ms)
2025-02-17 13:52:19.034 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842216853 ms)
2025-02-17 13:52:19.034 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842216854 ms)
2025-02-17 13:52:19.035 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34366)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842216854 ms)
2025-02-17 13:52:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842224412 ms)
2025-02-17 13:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842224413 ms)
2025-02-17 13:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:52:25.399999+00:00', 'runtime': '842220.001', 'adc_vb_f': '13.052', 'adc_va_f': '20.85', 'adc_vl_f': '13.043', 'adc_ic_f': '2.863', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.049', 'vb_ref': '14.355', 'ahc_r': '2631.3', 'ahc_t': '2631.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.354', 'sweep_vmp': '20.892', 'sweep_pmax': '36.327', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.052', 'ahc_daily': '1.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842224416 ms)
2025-02-17 13:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:51:25.400192+00:00', 'runtime': '842160.001', 'adc_vb_f': '13.049', 'adc_va_f': '20.85', 'adc_vl_f': '13.004', 'adc_ic_f': '2.793', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.04', 'vb_ref': '14.355', 'ahc_r': '2631.3', 'ahc_t': '2631.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.433', 'sweep_vmp': '20.892', 'sweep_pmax': '36.327', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.052', 'ahc_daily': '1.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842224417 ms)
2025-02-17 13:53:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842284402 ms)
2025-02-17 13:53:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842284402 ms)
2025-02-17 13:53:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:53:25.400167+00:00', 'runtime': '842280.001', 'adc_vb_f': '13.052', 'adc_va_f': '20.85', 'adc_vl_f': '13.043', 'adc_ic_f': '3.124', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.058', 'vb_ref': '14.355', 'ahc_r': '2631.4', 'ahc_t': '2631.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.827', 'sweep_vmp': '20.892', 'sweep_pmax': '36.327', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.086', 'ahc_daily': '1.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842284403 ms)
2025-02-17 13:53:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:52:25.399999+00:00', 'runtime': '842220.001', 'adc_vb_f': '13.052', 'adc_va_f': '20.85', 'adc_vl_f': '13.043', 'adc_ic_f': '2.863', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.049', 'vb_ref': '14.355', 'ahc_r': '2631.3', 'ahc_t': '2631.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.354', 'sweep_vmp': '20.892', 'sweep_pmax': '36.327', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.052', 'ahc_daily': '1.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842284404 ms)
2025-02-17 13:54:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842344406 ms)
2025-02-17 13:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842344407 ms)
2025-02-17 13:54:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:54:25.400271+00:00', 'runtime': '842340.001', 'adc_vb_f': '13.058', 'adc_va_f': '20.755', 'adc_vl_f': '13.052', 'adc_ic_f': '3.141', 'adc_il_f': '0.391', 't_hs': '7', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.055', 'vb_ref': '14.355', 'ahc_r': '2631.4', 'ahc_t': '2631.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.978', 'sweep_vmp': '20.798', 'sweep_pmax': '41.355', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.089', 'ahc_daily': '1.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842344408 ms)
2025-02-17 13:54:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:53:25.400167+00:00', 'runtime': '842280.001', 'adc_vb_f': '13.052', 'adc_va_f': '20.85', 'adc_vl_f': '13.043', 'adc_ic_f': '3.124', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.058', 'vb_ref': '14.355', 'ahc_r': '2631.4', 'ahc_t': '2631.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20120', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.827', 'sweep_vmp': '20.892', 'sweep_pmax': '36.327', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.086', 'ahc_daily': '1.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842344408 ms)
2025-02-17 13:54:28.283 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842346103 ms)
2025-02-17 13:54:28.284 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842346103 ms)
2025-02-17 13:54:28.284 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34500)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842346103 ms)
2025-02-17 13:54:55.392 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842373212 ms)
2025-02-17 13:54:55.393 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842373212 ms)
2025-02-17 13:54:55.393 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34532)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842373213 ms)
2025-02-17 13:55:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842404407 ms)
2025-02-17 13:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842404407 ms)
2025-02-17 13:55:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:55:25.400210+00:00', 'runtime': '842400.001', 'adc_vb_f': '13.092', 'adc_va_f': '20.755', 'adc_vl_f': '13.08', 'adc_ic_f': '3.298', 'adc_il_f': '0.331', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.083', 'vb_ref': '14.325', 'ahc_r': '2631.5', 'ahc_t': '2631.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.137', 'sweep_vmp': '20.798', 'sweep_pmax': '41.355', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.092', 'ahc_daily': '1.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842404408 ms)
2025-02-17 13:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:54:25.400271+00:00', 'runtime': '842340.001', 'adc_vb_f': '13.058', 'adc_va_f': '20.755', 'adc_vl_f': '13.052', 'adc_ic_f': '3.141', 'adc_il_f': '0.391', 't_hs': '7', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.055', 'vb_ref': '14.355', 'ahc_r': '2631.4', 'ahc_t': '2631.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.978', 'sweep_vmp': '20.798', 'sweep_pmax': '41.355', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.089', 'ahc_daily': '1.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842404409 ms)
2025-02-17 13:55:41.523 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842419343 ms)
2025-02-17 13:55:41.524 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842419343 ms)
2025-02-17 13:55:41.524 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34580)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842419343 ms)
2025-02-17 13:56:00.645 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842438464 ms)
2025-02-17 13:56:00.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842438464 ms)
2025-02-17 13:56:00.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34612)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842438465 ms)
2025-02-17 13:56:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842464412 ms)
2025-02-17 13:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842464413 ms)
2025-02-17 13:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:56:25.400181+00:00', 'runtime': '842460.001', 'adc_vb_f': '13.129', 'adc_va_f': '20.755', 'adc_vl_f': '13.12', 'adc_ic_f': '3.442', 'adc_il_f': '0.333', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.11', 'vb_ref': '14.325', 'ahc_r': '2631.5', 'ahc_t': '2631.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '45.115', 'sweep_vmp': '20.798', 'sweep_pmax': '41.355', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.129', 'ahc_daily': '1.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842464414 ms)
2025-02-17 13:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:55:25.400210+00:00', 'runtime': '842400.001', 'adc_vb_f': '13.092', 'adc_va_f': '20.755', 'adc_vl_f': '13.08', 'adc_ic_f': '3.298', 'adc_il_f': '0.331', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.083', 'vb_ref': '14.325', 'ahc_r': '2631.5', 'ahc_t': '2631.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.137', 'sweep_vmp': '20.798', 'sweep_pmax': '41.355', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.092', 'ahc_daily': '1.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842464414 ms)
2025-02-17 13:56:35.440 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+842473259 ms)
2025-02-17 13:56:35.440 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+842473259 ms)
2025-02-17 13:56:35.441 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:56:25.400336+00:00'} (T+842473261 ms)
2025-02-17 13:57:12.790 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842510610 ms)
2025-02-17 13:57:12.791 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842510610 ms)
2025-02-17 13:57:12.791 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34690)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842510610 ms)
2025-02-17 13:57:15.873 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842513692 ms)
2025-02-17 13:57:15.873 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842513693 ms)
2025-02-17 13:57:15.874 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34694)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842513693 ms)
2025-02-17 13:57:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842524414 ms)
2025-02-17 13:57:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842524414 ms)
2025-02-17 13:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:57:25.400188+00:00', 'runtime': '842520.001', 'adc_vb_f': '13.138', 'adc_va_f': '20.804', 'adc_vl_f': '13.12', 'adc_ic_f': '3.568', 'adc_il_f': '0.379', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.129', 'vb_ref': '14.325', 'ahc_r': '2631.6', 'ahc_t': '2631.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '46.821', 'sweep_vmp': '20.807', 'sweep_pmax': '46.02', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.156', 'ahc_daily': '1.6', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842524415 ms)
2025-02-17 13:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:56:25.400181+00:00', 'runtime': '842460.001', 'adc_vb_f': '13.129', 'adc_va_f': '20.755', 'adc_vl_f': '13.12', 'adc_ic_f': '3.442', 'adc_il_f': '0.333', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.11', 'vb_ref': '14.325', 'ahc_r': '2631.5', 'ahc_t': '2631.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '45.115', 'sweep_vmp': '20.798', 'sweep_pmax': '41.355', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.129', 'ahc_daily': '1.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842524416 ms)
2025-02-17 13:57:35.630 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+842533449 ms)
2025-02-17 13:57:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+842533449 ms)
2025-02-17 13:57:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 13:57:25.400382+00:00'} (T+842533450 ms)
2025-02-17 13:58:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842584415 ms)
2025-02-17 13:58:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842584416 ms)
2025-02-17 13:58:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:58:25.400183+00:00', 'runtime': '842580.001', 'adc_vb_f': '13.168', 'adc_va_f': '20.804', 'adc_vl_f': '13.156', 'adc_ic_f': '3.616', 'adc_il_f': '0.396', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.159', 'vb_ref': '14.325', 'ahc_r': '2631.6', 'ahc_t': '2631.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '47.621', 'sweep_vmp': '20.807', 'sweep_pmax': '46.02', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.168', 'ahc_daily': '1.6', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842584417 ms)
2025-02-17 13:58:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:57:25.400188+00:00', 'runtime': '842520.001', 'adc_vb_f': '13.138', 'adc_va_f': '20.804', 'adc_vl_f': '13.12', 'adc_ic_f': '3.568', 'adc_il_f': '0.379', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.129', 'vb_ref': '14.325', 'ahc_r': '2631.6', 'ahc_t': '2631.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '46.821', 'sweep_vmp': '20.807', 'sweep_pmax': '46.02', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.156', 'ahc_daily': '1.6', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842584417 ms)
2025-02-17 13:59:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842644417 ms)
2025-02-17 13:59:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842644417 ms)
2025-02-17 13:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 13:59:25.400256+00:00', 'runtime': '842640.001', 'adc_vb_f': '13.168', 'adc_va_f': '20.804', 'adc_vl_f': '13.156', 'adc_ic_f': '3.667', 'adc_il_f': '0.394', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.168', 'vb_ref': '14.325', 'ahc_r': '2631.7', 'ahc_t': '2631.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '48.255', 'sweep_vmp': '20.807', 'sweep_pmax': '46.02', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.168', 'ahc_daily': '1.7', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842644418 ms)
2025-02-17 13:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:58:25.400183+00:00', 'runtime': '842580.001', 'adc_vb_f': '13.168', 'adc_va_f': '20.804', 'adc_vl_f': '13.156', 'adc_ic_f': '3.616', 'adc_il_f': '0.396', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.159', 'vb_ref': '14.325', 'ahc_r': '2631.6', 'ahc_t': '2631.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '47.621', 'sweep_vmp': '20.807', 'sweep_pmax': '46.02', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.168', 'ahc_daily': '1.6', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842644419 ms)
2025-02-17 13:59:28.105 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842645924 ms)
2025-02-17 13:59:28.105 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842645924 ms)
2025-02-17 13:59:28.105 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34836)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842645925 ms)
2025-02-17 13:59:50.173 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842667993 ms)
2025-02-17 13:59:50.174 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842667993 ms)
2025-02-17 13:59:50.174 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34862)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842667993 ms)
2025-02-17 14:00:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842704405 ms)
2025-02-17 14:00:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842704405 ms)
2025-02-17 14:00:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:00:25.400162+00:00', 'runtime': '842700.001', 'adc_vb_f': '13.202', 'adc_va_f': '20.85', 'adc_vl_f': '13.196', 'adc_ic_f': '3.805', 'adc_il_f': '0.367', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.184', 'vb_ref': '14.325', 'ahc_r': '2631.8', 'ahc_t': '2631.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '50.218', 'sweep_vmp': '20.853', 'sweep_pmax': '48.844', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '13.205', 'ahc_daily': '1.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842704406 ms)
2025-02-17 14:00:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 13:59:25.400256+00:00', 'runtime': '842640.001', 'adc_vb_f': '13.168', 'adc_va_f': '20.804', 'adc_vl_f': '13.156', 'adc_ic_f': '3.667', 'adc_il_f': '0.394', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.168', 'vb_ref': '14.325', 'ahc_r': '2631.7', 'ahc_t': '2631.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.5', 'ahl_t': '1590.5', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '48.255', 'sweep_vmp': '20.807', 'sweep_pmax': '46.02', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.168', 'ahc_daily': '1.7', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842704406 ms)
2025-02-17 14:00:35.491 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+842713310 ms)
2025-02-17 14:00:35.491 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+842713311 ms)
2025-02-17 14:00:35.492 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:00:25.400328+00:00'} (T+842713312 ms)
2025-02-17 14:01:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842764408 ms)
2025-02-17 14:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842764408 ms)
2025-02-17 14:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:01:25.400201+00:00', 'runtime': '842760.001', 'adc_vb_f': '13.205', 'adc_va_f': '20.853', 'adc_vl_f': '13.196', 'adc_ic_f': '3.829', 'adc_il_f': '0.387', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.202', 'vb_ref': '14.325', 'ahc_r': '2631.8', 'ahc_t': '2631.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '50.656', 'sweep_vmp': '20.853', 'sweep_pmax': '48.844', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '13.205', 'ahc_daily': '1.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842764410 ms)
2025-02-17 14:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:00:25.400162+00:00', 'runtime': '842700.001', 'adc_vb_f': '13.202', 'adc_va_f': '20.85', 'adc_vl_f': '13.196', 'adc_ic_f': '3.805', 'adc_il_f': '0.367', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.184', 'vb_ref': '14.325', 'ahc_r': '2631.8', 'ahc_t': '2631.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '50.218', 'sweep_vmp': '20.853', 'sweep_pmax': '48.844', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '13.205', 'ahc_daily': '1.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842764411 ms)
2025-02-17 14:02:20.434 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+842818253 ms)
2025-02-17 14:02:20.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+842818253 ms)
2025-02-17 14:02:20.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35022)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+842818254 ms)
2025-02-17 14:02:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842824411 ms)
2025-02-17 14:02:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842824412 ms)
2025-02-17 14:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:02:25.400193+00:00', 'runtime': '842820.001', 'adc_vb_f': '13.208', 'adc_va_f': '20.85', 'adc_vl_f': '13.232', 'adc_ic_f': '3.872', 'adc_il_f': '0.355', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.205', 'vb_ref': '14.325', 'ahc_r': '2631.9', 'ahc_t': '2631.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '51.124', 'sweep_vmp': '20.853', 'sweep_pmax': '48.844', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '13.211', 'ahc_daily': '1.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842824413 ms)
2025-02-17 14:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:01:25.400201+00:00', 'runtime': '842760.001', 'adc_vb_f': '13.205', 'adc_va_f': '20.853', 'adc_vl_f': '13.196', 'adc_ic_f': '3.829', 'adc_il_f': '0.387', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.202', 'vb_ref': '14.325', 'ahc_r': '2631.8', 'ahc_t': '2631.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '50.656', 'sweep_vmp': '20.853', 'sweep_pmax': '48.844', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '13.205', 'ahc_daily': '1.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842824413 ms)
2025-02-17 14:03:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842884415 ms)
2025-02-17 14:03:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842884416 ms)
2025-02-17 14:03:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:03:25.400176+00:00', 'runtime': '842880.001', 'adc_vb_f': '13.22', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.882', 'adc_il_f': '0.362', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.211', 'vb_ref': '14.296', 'ahc_r': '2632.0', 'ahc_t': '2632.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '51.35', 'sweep_vmp': '20.911', 'sweep_pmax': '51.199', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842884417 ms)
2025-02-17 14:03:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:02:25.400193+00:00', 'runtime': '842820.001', 'adc_vb_f': '13.208', 'adc_va_f': '20.85', 'adc_vl_f': '13.232', 'adc_ic_f': '3.872', 'adc_il_f': '0.355', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.205', 'vb_ref': '14.325', 'ahc_r': '2631.9', 'ahc_t': '2631.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '51.124', 'sweep_vmp': '20.853', 'sweep_pmax': '48.844', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '13.211', 'ahc_daily': '1.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842884417 ms)
2025-02-17 14:04:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+842944402 ms)
2025-02-17 14:04:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+842944402 ms)
2025-02-17 14:04:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:04:25.400260+00:00', 'runtime': '842940.001', 'adc_vb_f': '13.208', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.817', 'adc_il_f': '0.37', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.208', 'vb_ref': '14.296', 'ahc_r': '2632.0', 'ahc_t': '2632.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '50.399', 'sweep_vmp': '20.911', 'sweep_pmax': '51.199', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842944405 ms)
2025-02-17 14:04:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:03:25.400176+00:00', 'runtime': '842880.001', 'adc_vb_f': '13.22', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.882', 'adc_il_f': '0.362', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.211', 'vb_ref': '14.296', 'ahc_r': '2632.0', 'ahc_t': '2632.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '51.35', 'sweep_vmp': '20.911', 'sweep_pmax': '51.199', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+842944405 ms)
2025-02-17 14:05:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843004406 ms)
2025-02-17 14:05:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843004406 ms)
2025-02-17 14:05:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:05:25.400193+00:00', 'runtime': '843000.001', 'adc_vb_f': '13.205', 'adc_va_f': '20.898', 'adc_vl_f': '13.217', 'adc_ic_f': '3.778', 'adc_il_f': '0.343', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.205', 'vb_ref': '14.296', 'ahc_r': '2632.1', 'ahc_t': '2632.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '49.855', 'sweep_vmp': '20.911', 'sweep_pmax': '51.199', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.1', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843004409 ms)
2025-02-17 14:05:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:04:25.400260+00:00', 'runtime': '842940.001', 'adc_vb_f': '13.208', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.817', 'adc_il_f': '0.37', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.208', 'vb_ref': '14.296', 'ahc_r': '2632.0', 'ahc_t': '2632.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '50.399', 'sweep_vmp': '20.911', 'sweep_pmax': '51.199', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843004409 ms)
2025-02-17 14:05:35.910 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843013729 ms)
2025-02-17 14:05:35.910 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843013730 ms)
2025-02-17 14:05:35.911 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:05:25.400335+00:00'} (T+843013730 ms)
2025-02-17 14:05:39.770 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843017589 ms)
2025-02-17 14:05:39.770 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843017590 ms)
2025-02-17 14:05:39.771 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35232)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843017590 ms)
2025-02-17 14:06:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843064407 ms)
2025-02-17 14:06:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843064408 ms)
2025-02-17 14:06:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:06:25.400175+00:00', 'runtime': '843060.001', 'adc_vb_f': '13.242', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.926', 'adc_il_f': '0.365', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.226', 'vb_ref': '14.296', 'ahc_r': '2632.1', 'ahc_t': '2632.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '51.879', 'sweep_vmp': '20.911', 'sweep_pmax': '51.486', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.1', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843064409 ms)
2025-02-17 14:06:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:05:25.400193+00:00', 'runtime': '843000.001', 'adc_vb_f': '13.205', 'adc_va_f': '20.898', 'adc_vl_f': '13.217', 'adc_ic_f': '3.778', 'adc_il_f': '0.343', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.205', 'vb_ref': '14.296', 'ahc_r': '2632.1', 'ahc_t': '2632.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '49.855', 'sweep_vmp': '20.911', 'sweep_pmax': '51.199', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.1', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843064409 ms)
2025-02-17 14:06:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843073431 ms)
2025-02-17 14:06:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843073432 ms)
2025-02-17 14:06:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:06:25.400407+00:00'} (T+843073433 ms)
2025-02-17 14:07:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843124411 ms)
2025-02-17 14:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843124412 ms)
2025-02-17 14:07:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:07:25.400200+00:00', 'runtime': '843120.001', 'adc_vb_f': '13.245', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.955', 'adc_il_f': '0.355', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.242', 'vb_ref': '14.296', 'ahc_r': '2632.2', 'ahc_t': '2632.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '52.377', 'sweep_vmp': '20.911', 'sweep_pmax': '51.486', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.2', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843124413 ms)
2025-02-17 14:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:06:25.400175+00:00', 'runtime': '843060.001', 'adc_vb_f': '13.242', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.926', 'adc_il_f': '0.365', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.226', 'vb_ref': '14.296', 'ahc_r': '2632.1', 'ahc_t': '2632.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '51.879', 'sweep_vmp': '20.911', 'sweep_pmax': '51.486', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.1', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843124413 ms)
2025-02-17 14:08:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843184413 ms)
2025-02-17 14:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843184414 ms)
2025-02-17 14:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:08:25.400197+00:00', 'runtime': '843180.001', 'adc_vb_f': '13.245', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.952', 'adc_il_f': '0.401', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.245', 'vb_ref': '14.296', 'ahc_r': '2632.3', 'ahc_t': '2632.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '52.332', 'sweep_vmp': '20.911', 'sweep_pmax': '51.486', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.3', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843184415 ms)
2025-02-17 14:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:07:25.400200+00:00', 'runtime': '843120.001', 'adc_vb_f': '13.245', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.955', 'adc_il_f': '0.355', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.242', 'vb_ref': '14.296', 'ahc_r': '2632.2', 'ahc_t': '2632.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '52.377', 'sweep_vmp': '20.911', 'sweep_pmax': '51.486', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.2', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843184415 ms)
2025-02-17 14:09:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843244418 ms)
2025-02-17 14:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843244419 ms)
2025-02-17 14:09:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:09:25.400156+00:00', 'runtime': '843240.001', 'adc_vb_f': '13.245', 'adc_va_f': '20.944', 'adc_vl_f': '13.232', 'adc_ic_f': '3.935', 'adc_il_f': '0.396', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.245', 'vb_ref': '14.296', 'ahc_r': '2632.3', 'ahc_t': '2632.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '52.075', 'sweep_vmp': '20.963', 'sweep_pmax': '52.573', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.272', 'ahc_daily': '2.3', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843244420 ms)
2025-02-17 14:09:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:08:25.400197+00:00', 'runtime': '843180.001', 'adc_vb_f': '13.245', 'adc_va_f': '20.898', 'adc_vl_f': '13.232', 'adc_ic_f': '3.952', 'adc_il_f': '0.401', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.245', 'vb_ref': '14.296', 'ahc_r': '2632.3', 'ahc_t': '2632.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '52.332', 'sweep_vmp': '20.911', 'sweep_pmax': '51.486', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.245', 'ahc_daily': '2.3', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843244420 ms)
2025-02-17 14:09:35.835 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843253655 ms)
2025-02-17 14:09:35.836 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843253656 ms)
2025-02-17 14:09:35.839 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:09:25.400330+00:00'} (T+843253658 ms)
2025-02-17 14:09:53.156 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843270975 ms)
2025-02-17 14:09:53.156 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843270975 ms)
2025-02-17 14:09:53.157 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35512)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843270976 ms)
2025-02-17 14:10:23.278 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843301097 ms)
2025-02-17 14:10:23.278 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843301097 ms)
2025-02-17 14:10:23.278 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35544)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843301098 ms)
2025-02-17 14:10:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843304404 ms)
2025-02-17 14:10:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843304405 ms)
2025-02-17 14:10:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:10:25.400193+00:00', 'runtime': '843300.001', 'adc_vb_f': '13.26', 'adc_va_f': '20.944', 'adc_vl_f': '13.263', 'adc_ic_f': '4.051', 'adc_il_f': '0.387', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.254', 'vb_ref': '14.266', 'ahc_r': '2632.4', 'ahc_t': '2632.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '53.751', 'sweep_vmp': '20.963', 'sweep_pmax': '52.573', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.272', 'ahc_daily': '2.4', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843304406 ms)
2025-02-17 14:10:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:09:25.400156+00:00', 'runtime': '843240.001', 'adc_vb_f': '13.245', 'adc_va_f': '20.944', 'adc_vl_f': '13.232', 'adc_ic_f': '3.935', 'adc_il_f': '0.396', 't_hs': '9', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.245', 'vb_ref': '14.296', 'ahc_r': '2632.3', 'ahc_t': '2632.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '52.075', 'sweep_vmp': '20.963', 'sweep_pmax': '52.573', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.272', 'ahc_daily': '2.3', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843304406 ms)
2025-02-17 14:11:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843364410 ms)
2025-02-17 14:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843364411 ms)
2025-02-17 14:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:11:25.400205+00:00', 'runtime': '843360.001', 'adc_vb_f': '13.281', 'adc_va_f': '20.944', 'adc_vl_f': '13.272', 'adc_ic_f': '4.189', 'adc_il_f': '0.377', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.275', 'vb_ref': '14.266', 'ahc_r': '2632.5', 'ahc_t': '2632.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '55.593', 'sweep_vmp': '20.963', 'sweep_pmax': '52.573', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.281', 'ahc_daily': '2.5', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843364412 ms)
2025-02-17 14:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:10:25.400193+00:00', 'runtime': '843300.001', 'adc_vb_f': '13.26', 'adc_va_f': '20.944', 'adc_vl_f': '13.263', 'adc_ic_f': '4.051', 'adc_il_f': '0.387', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.254', 'vb_ref': '14.266', 'ahc_r': '2632.4', 'ahc_t': '2632.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '53.751', 'sweep_vmp': '20.963', 'sweep_pmax': '52.573', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.272', 'ahc_daily': '2.4', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843364412 ms)
2025-02-17 14:11:57.463 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843395282 ms)
2025-02-17 14:11:57.463 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843395282 ms)
2025-02-17 14:11:57.464 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35646)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843395283 ms)
2025-02-17 14:12:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843424415 ms)
2025-02-17 14:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843424416 ms)
2025-02-17 14:12:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:12:25.400163+00:00', 'runtime': '843420.001', 'adc_vb_f': '13.321', 'adc_va_f': '20.944', 'adc_vl_f': '13.309', 'adc_ic_f': '4.377', 'adc_il_f': '0.326', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.303', 'vb_ref': '14.266', 'ahc_r': '2632.6', 'ahc_t': '2632.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '58.341', 'sweep_vmp': '20.975', 'sweep_pmax': '57.39', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.321', 'ahc_daily': '2.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843424417 ms)
2025-02-17 14:12:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:11:25.400205+00:00', 'runtime': '843360.001', 'adc_vb_f': '13.281', 'adc_va_f': '20.944', 'adc_vl_f': '13.272', 'adc_ic_f': '4.189', 'adc_il_f': '0.377', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.275', 'vb_ref': '14.266', 'ahc_r': '2632.5', 'ahc_t': '2632.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '55.593', 'sweep_vmp': '20.963', 'sweep_pmax': '52.573', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.281', 'ahc_daily': '2.5', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843424418 ms)
2025-02-17 14:12:36.984 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843434803 ms)
2025-02-17 14:12:36.985 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843434804 ms)
2025-02-17 14:12:36.986 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:12:25.400338+00:00'} (T+843434805 ms)
2025-02-17 14:13:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843484420 ms)
2025-02-17 14:13:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843484420 ms)
2025-02-17 14:13:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:13:25.400162+00:00', 'runtime': '843480.001', 'adc_vb_f': '13.327', 'adc_va_f': '20.944', 'adc_vl_f': '13.348', 'adc_ic_f': '4.46', 'adc_il_f': '0.324', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.321', 'vb_ref': '14.266', 'ahc_r': '2632.6', 'ahc_t': '2632.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '59.337', 'sweep_vmp': '20.975', 'sweep_pmax': '57.39', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.33', 'ahc_daily': '2.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843484421 ms)
2025-02-17 14:13:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:12:25.400163+00:00', 'runtime': '843420.001', 'adc_vb_f': '13.321', 'adc_va_f': '20.944', 'adc_vl_f': '13.309', 'adc_ic_f': '4.377', 'adc_il_f': '0.326', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.303', 'vb_ref': '14.266', 'ahc_r': '2632.6', 'ahc_t': '2632.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '58.341', 'sweep_vmp': '20.975', 'sweep_pmax': '57.39', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.321', 'ahc_daily': '2.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843484422 ms)
2025-02-17 14:13:35.902 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843493721 ms)
2025-02-17 14:13:35.903 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843493722 ms)
2025-02-17 14:13:35.904 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:13:25.400344+00:00'} (T+843493723 ms)
2025-02-17 14:14:03.692 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843521512 ms)
2025-02-17 14:14:03.693 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843521512 ms)
2025-02-17 14:14:03.693 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35782)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843521512 ms)
2025-02-17 14:14:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843544409 ms)
2025-02-17 14:14:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843544409 ms)
2025-02-17 14:14:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:14:25.400148+00:00', 'runtime': '843540.001', 'adc_vb_f': '13.354', 'adc_va_f': '20.944', 'adc_vl_f': '13.348', 'adc_ic_f': '4.544', 'adc_il_f': '0.331', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.339', 'vb_ref': '14.266', 'ahc_r': '2632.7', 'ahc_t': '2632.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '60.696', 'sweep_vmp': '20.975', 'sweep_pmax': '57.39', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.358', 'ahc_daily': '2.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843544410 ms)
2025-02-17 14:14:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:13:25.400162+00:00', 'runtime': '843480.001', 'adc_vb_f': '13.327', 'adc_va_f': '20.944', 'adc_vl_f': '13.348', 'adc_ic_f': '4.46', 'adc_il_f': '0.324', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.321', 'vb_ref': '14.266', 'ahc_r': '2632.6', 'ahc_t': '2632.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '59.337', 'sweep_vmp': '20.975', 'sweep_pmax': '57.39', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.33', 'ahc_daily': '2.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843544411 ms)
2025-02-17 14:14:35.832 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843553651 ms)
2025-02-17 14:14:35.832 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843553652 ms)
2025-02-17 14:14:35.833 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:14:25.400402+00:00'} (T+843553653 ms)
2025-02-17 14:14:50.821 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843568640 ms)
2025-02-17 14:14:50.822 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843568641 ms)
2025-02-17 14:14:50.822 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35834)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843568641 ms)
2025-02-17 14:15:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843604415 ms)
2025-02-17 14:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843604416 ms)
2025-02-17 14:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:15:25.400166+00:00', 'runtime': '843600.001', 'adc_vb_f': '13.358', 'adc_va_f': '20.944', 'adc_vl_f': '13.351', 'adc_ic_f': '4.609', 'adc_il_f': '0.348', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.354', 'vb_ref': '14.266', 'ahc_r': '2632.8', 'ahc_t': '2632.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '61.496', 'sweep_vmp': '20.978', 'sweep_pmax': '61.315', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.367', 'ahc_daily': '2.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843604417 ms)
2025-02-17 14:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:14:25.400148+00:00', 'runtime': '843540.001', 'adc_vb_f': '13.354', 'adc_va_f': '20.944', 'adc_vl_f': '13.348', 'adc_ic_f': '4.544', 'adc_il_f': '0.331', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.339', 'vb_ref': '14.266', 'ahc_r': '2632.7', 'ahc_t': '2632.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '60.696', 'sweep_vmp': '20.975', 'sweep_pmax': '57.39', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.358', 'ahc_daily': '2.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843604417 ms)
2025-02-17 14:15:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843613420 ms)
2025-02-17 14:15:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843613420 ms)
2025-02-17 14:15:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:15:25.399991+00:00'} (T+843613421 ms)
2025-02-17 14:16:20.030 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843657849 ms)
2025-02-17 14:16:20.030 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843657849 ms)
2025-02-17 14:16:20.031 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35938)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843657850 ms)
2025-02-17 14:16:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843664402 ms)
2025-02-17 14:16:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843664402 ms)
2025-02-17 14:16:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:16:25.400156+00:00', 'runtime': '843660.001', 'adc_vb_f': '13.358', 'adc_va_f': '20.944', 'adc_vl_f': '13.385', 'adc_ic_f': '4.696', 'adc_il_f': '0.374', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.358', 'vb_ref': '14.266', 'ahc_r': '2632.9', 'ahc_t': '2632.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '62.719', 'sweep_vmp': '20.978', 'sweep_pmax': '61.315', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.367', 'ahc_daily': '2.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843664403 ms)
2025-02-17 14:16:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:15:25.400166+00:00', 'runtime': '843600.001', 'adc_vb_f': '13.358', 'adc_va_f': '20.944', 'adc_vl_f': '13.351', 'adc_ic_f': '4.609', 'adc_il_f': '0.348', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.354', 'vb_ref': '14.266', 'ahc_r': '2632.8', 'ahc_t': '2632.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '61.496', 'sweep_vmp': '20.978', 'sweep_pmax': '61.315', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.367', 'ahc_daily': '2.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843664404 ms)
2025-02-17 14:16:35.646 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843673465 ms)
2025-02-17 14:16:35.647 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843673466 ms)
2025-02-17 14:16:35.648 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:16:25.400182+00:00'} (T+843673467 ms)
2025-02-17 14:16:45.151 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843682971 ms)
2025-02-17 14:16:45.152 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843682971 ms)
2025-02-17 14:16:45.152 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35966)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843682972 ms)
2025-02-17 14:17:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843724407 ms)
2025-02-17 14:17:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843724407 ms)
2025-02-17 14:17:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:17:25.400186+00:00', 'runtime': '843720.001', 'adc_vb_f': '13.382', 'adc_va_f': '20.944', 'adc_vl_f': '13.385', 'adc_ic_f': '4.752', 'adc_il_f': '0.374', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.376', 'vb_ref': '14.266', 'ahc_r': '2632.9', 'ahc_t': '2632.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '63.535', 'sweep_vmp': '20.978', 'sweep_pmax': '61.315', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.397', 'ahc_daily': '2.9', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843724408 ms)
2025-02-17 14:17:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:16:25.400156+00:00', 'runtime': '843660.001', 'adc_vb_f': '13.358', 'adc_va_f': '20.944', 'adc_vl_f': '13.385', 'adc_ic_f': '4.696', 'adc_il_f': '0.374', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.358', 'vb_ref': '14.266', 'ahc_r': '2632.9', 'ahc_t': '2632.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.6', 'ahl_t': '1590.6', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '62.719', 'sweep_vmp': '20.978', 'sweep_pmax': '61.315', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.367', 'ahc_daily': '2.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843724409 ms)
2025-02-17 14:17:35.799 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843733619 ms)
2025-02-17 14:17:35.800 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843733619 ms)
2025-02-17 14:17:35.801 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:17:25.400186+00:00'} (T+843733620 ms)
2025-02-17 14:18:25.280 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+843783099 ms)
2025-02-17 14:18:25.280 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+843783099 ms)
2025-02-17 14:18:25.280 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843783100 ms)
2025-02-17 14:18:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843784414 ms)
2025-02-17 14:18:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843784414 ms)
2025-02-17 14:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:18:25.400208+00:00', 'runtime': '843780.001', 'adc_vb_f': '13.397', 'adc_va_f': '20.944', 'adc_vl_f': '13.385', 'adc_ic_f': '4.793', 'adc_il_f': '0.379', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.391', 'vb_ref': '14.266', 'ahc_r': '2633.0', 'ahc_t': '2633.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '64.305', 'sweep_vmp': '20.987', 'sweep_pmax': '63.64', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.4', 'ahc_daily': '3.0', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843784415 ms)
2025-02-17 14:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:17:25.400186+00:00', 'runtime': '843720.001', 'adc_vb_f': '13.382', 'adc_va_f': '20.944', 'adc_vl_f': '13.385', 'adc_ic_f': '4.752', 'adc_il_f': '0.374', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.376', 'vb_ref': '14.266', 'ahc_r': '2632.9', 'ahc_t': '2632.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '63.535', 'sweep_vmp': '20.978', 'sweep_pmax': '61.315', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.397', 'ahc_daily': '2.9', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843784416 ms)
2025-02-17 14:18:35.739 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843793558 ms)
2025-02-17 14:18:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+843793559 ms)
2025-02-17 14:18:35.740 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:18:25.400023+00:00'} (T+843793560 ms)
2025-02-17 14:19:22.592 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+843840412 ms)
2025-02-17 14:19:22.593 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+843840412 ms)
2025-02-17 14:19:22.593 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36134)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+843840412 ms)
2025-02-17 14:19:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843844405 ms)
2025-02-17 14:19:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843844405 ms)
2025-02-17 14:19:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:19:25.400158+00:00', 'runtime': '843840.001', 'adc_vb_f': '13.4', 'adc_va_f': '20.944', 'adc_vl_f': '13.419', 'adc_ic_f': '4.856', 'adc_il_f': '0.329', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.397', 'vb_ref': '14.266', 'ahc_r': '2633.1', 'ahc_t': '2633.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '65.105', 'sweep_vmp': '20.987', 'sweep_pmax': '63.64', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.403', 'ahc_daily': '3.1', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843844406 ms)
2025-02-17 14:19:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:18:25.400208+00:00', 'runtime': '843780.001', 'adc_vb_f': '13.397', 'adc_va_f': '20.944', 'adc_vl_f': '13.385', 'adc_ic_f': '4.793', 'adc_il_f': '0.379', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.391', 'vb_ref': '14.266', 'ahc_r': '2633.0', 'ahc_t': '2633.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '64.305', 'sweep_vmp': '20.987', 'sweep_pmax': '63.64', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.4', 'ahc_daily': '3.0', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843844407 ms)
2025-02-17 14:20:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843904410 ms)
2025-02-17 14:20:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843904410 ms)
2025-02-17 14:20:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:20:25.400166+00:00', 'runtime': '843900.001', 'adc_vb_f': '13.428', 'adc_va_f': '20.944', 'adc_vl_f': '13.425', 'adc_ic_f': '4.906', 'adc_il_f': '0.312', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.412', 'vb_ref': '14.266', 'ahc_r': '2633.2', 'ahc_t': '2633.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '65.86', 'sweep_vmp': '20.987', 'sweep_pmax': '63.64', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.428', 'ahc_daily': '3.2', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843904411 ms)
2025-02-17 14:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:19:25.400158+00:00', 'runtime': '843840.001', 'adc_vb_f': '13.4', 'adc_va_f': '20.944', 'adc_vl_f': '13.419', 'adc_ic_f': '4.856', 'adc_il_f': '0.329', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.397', 'vb_ref': '14.266', 'ahc_r': '2633.1', 'ahc_t': '2633.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '65.105', 'sweep_vmp': '20.987', 'sweep_pmax': '63.64', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.403', 'ahc_daily': '3.1', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843904414 ms)
2025-02-17 14:21:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+843964415 ms)
2025-02-17 14:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+843964415 ms)
2025-02-17 14:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:21:25.400265+00:00', 'runtime': '843960.001', 'adc_vb_f': '13.434', 'adc_va_f': '20.944', 'adc_vl_f': '13.425', 'adc_ic_f': '4.962', 'adc_il_f': '0.336', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.428', 'vb_ref': '14.266', 'ahc_r': '2633.3', 'ahc_t': '2633.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '66.645', 'sweep_vmp': '20.987', 'sweep_pmax': '66.102', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.434', 'ahc_daily': '3.3', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843964416 ms)
2025-02-17 14:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:20:25.400166+00:00', 'runtime': '843900.001', 'adc_vb_f': '13.428', 'adc_va_f': '20.944', 'adc_vl_f': '13.425', 'adc_ic_f': '4.906', 'adc_il_f': '0.312', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.412', 'vb_ref': '14.266', 'ahc_r': '2633.2', 'ahc_t': '2633.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '65.86', 'sweep_vmp': '20.987', 'sweep_pmax': '63.64', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.428', 'ahc_daily': '3.2', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+843964417 ms)
2025-02-17 14:22:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844024404 ms)
2025-02-17 14:22:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844024405 ms)
2025-02-17 14:22:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:22:25.400225+00:00', 'runtime': '844020.001', 'adc_vb_f': '13.434', 'adc_va_f': '20.944', 'adc_vl_f': '13.437', 'adc_ic_f': '4.989', 'adc_il_f': '0.329', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.434', 'vb_ref': '14.266', 'ahc_r': '2633.3', 'ahc_t': '2633.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '67.068', 'sweep_vmp': '20.987', 'sweep_pmax': '66.102', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.434', 'ahc_daily': '3.3', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844024407 ms)
2025-02-17 14:22:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:21:25.400265+00:00', 'runtime': '843960.001', 'adc_vb_f': '13.434', 'adc_va_f': '20.944', 'adc_vl_f': '13.425', 'adc_ic_f': '4.962', 'adc_il_f': '0.336', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.428', 'vb_ref': '14.266', 'ahc_r': '2633.3', 'ahc_t': '2633.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '66.645', 'sweep_vmp': '20.987', 'sweep_pmax': '66.102', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.434', 'ahc_daily': '3.3', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844024407 ms)
2025-02-17 14:22:56.888 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+844054707 ms)
2025-02-17 14:22:56.888 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+844054708 ms)
2025-02-17 14:22:56.889 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36360)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844054708 ms)
2025-02-17 14:23:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844084423 ms)
2025-02-17 14:23:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844084423 ms)
2025-02-17 14:23:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:23:25.400187+00:00', 'runtime': '844080.001', 'adc_vb_f': '13.44', 'adc_va_f': '20.944', 'adc_vl_f': '13.461', 'adc_ic_f': '5.044', 'adc_il_f': '0.348', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.437', 'vb_ref': '14.266', 'ahc_r': '2633.4', 'ahc_t': '2633.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '67.823', 'sweep_vmp': '20.987', 'sweep_pmax': '66.102', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.452', 'ahc_daily': '3.4', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844084427 ms)
2025-02-17 14:23:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:22:25.400225+00:00', 'runtime': '844020.001', 'adc_vb_f': '13.434', 'adc_va_f': '20.944', 'adc_vl_f': '13.437', 'adc_ic_f': '4.989', 'adc_il_f': '0.329', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.434', 'vb_ref': '14.266', 'ahc_r': '2633.3', 'ahc_t': '2633.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '67.068', 'sweep_vmp': '20.987', 'sweep_pmax': '66.102', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.434', 'ahc_daily': '3.3', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844084427 ms)
2025-02-17 14:24:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844144415 ms)
2025-02-17 14:24:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844144415 ms)
2025-02-17 14:24:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:24:25.400193+00:00', 'runtime': '844140.001', 'adc_vb_f': '13.455', 'adc_va_f': '20.898', 'adc_vl_f': '13.461', 'adc_ic_f': '5.066', 'adc_il_f': '0.331', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.446', 'vb_ref': '14.266', 'ahc_r': '2633.5', 'ahc_t': '2633.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '68.215', 'sweep_vmp': '20.935', 'sweep_pmax': '67.823', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.461', 'ahc_daily': '3.5', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844144420 ms)
2025-02-17 14:24:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:23:25.400187+00:00', 'runtime': '844080.001', 'adc_vb_f': '13.44', 'adc_va_f': '20.944', 'adc_vl_f': '13.461', 'adc_ic_f': '5.044', 'adc_il_f': '0.348', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.437', 'vb_ref': '14.266', 'ahc_r': '2633.4', 'ahc_t': '2633.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '67.823', 'sweep_vmp': '20.987', 'sweep_pmax': '66.102', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.452', 'ahc_daily': '3.4', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844144420 ms)
2025-02-17 14:25:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844204403 ms)
2025-02-17 14:25:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844204404 ms)
2025-02-17 14:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:25:25.400168+00:00', 'runtime': '844200.001', 'adc_vb_f': '13.47', 'adc_va_f': '20.898', 'adc_vl_f': '13.461', 'adc_ic_f': '5.126', 'adc_il_f': '0.362', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.464', 'vb_ref': '14.266', 'ahc_r': '2633.6', 'ahc_t': '2633.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '68.985', 'sweep_vmp': '20.935', 'sweep_pmax': '67.823', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.47', 'ahc_daily': '3.6', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844204408 ms)
2025-02-17 14:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:24:25.400193+00:00', 'runtime': '844140.001', 'adc_vb_f': '13.455', 'adc_va_f': '20.898', 'adc_vl_f': '13.461', 'adc_ic_f': '5.066', 'adc_il_f': '0.331', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.446', 'vb_ref': '14.266', 'ahc_r': '2633.5', 'ahc_t': '2633.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '68.215', 'sweep_vmp': '20.935', 'sweep_pmax': '67.823', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.461', 'ahc_daily': '3.5', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844204408 ms)
2025-02-17 14:26:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844264408 ms)
2025-02-17 14:26:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844264409 ms)
2025-02-17 14:26:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:26:25.400168+00:00', 'runtime': '844260.001', 'adc_vb_f': '13.47', 'adc_va_f': '20.895', 'adc_vl_f': '13.467', 'adc_ic_f': '5.158', 'adc_il_f': '0.379', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.47', 'vb_ref': '14.266', 'ahc_r': '2633.7', 'ahc_t': '2633.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '69.469', 'sweep_vmp': '20.935', 'sweep_pmax': '67.823', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.47', 'ahc_daily': '3.7', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844264410 ms)
2025-02-17 14:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:25:25.400168+00:00', 'runtime': '844200.001', 'adc_vb_f': '13.47', 'adc_va_f': '20.898', 'adc_vl_f': '13.461', 'adc_ic_f': '5.126', 'adc_il_f': '0.362', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.464', 'vb_ref': '14.266', 'ahc_r': '2633.6', 'ahc_t': '2633.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '68.985', 'sweep_vmp': '20.935', 'sweep_pmax': '67.823', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.47', 'ahc_daily': '3.6', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844264410 ms)
2025-02-17 14:26:34.215 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+844272034 ms)
2025-02-17 14:26:34.215 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+844272035 ms)
2025-02-17 14:26:34.216 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36596)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844272035 ms)
2025-02-17 14:27:21.376 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+844319196 ms)
2025-02-17 14:27:21.377 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+844319196 ms)
2025-02-17 14:27:21.377 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36648)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844319196 ms)
2025-02-17 14:27:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844324415 ms)
2025-02-17 14:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844324416 ms)
2025-02-17 14:27:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:27:25.400189+00:00', 'runtime': '844320.001', 'adc_vb_f': '13.492', 'adc_va_f': '20.944', 'adc_vl_f': '13.501', 'adc_ic_f': '5.225', 'adc_il_f': '0.329', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.477', 'vb_ref': '14.237', 'ahc_r': '2633.8', 'ahc_t': '2633.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '70.541', 'sweep_vmp': '20.993', 'sweep_pmax': '70.118', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.498', 'ahc_daily': '3.8', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844324417 ms)
2025-02-17 14:27:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:26:25.400168+00:00', 'runtime': '844260.001', 'adc_vb_f': '13.47', 'adc_va_f': '20.895', 'adc_vl_f': '13.467', 'adc_ic_f': '5.158', 'adc_il_f': '0.379', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.47', 'vb_ref': '14.266', 'ahc_r': '2633.7', 'ahc_t': '2633.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '69.469', 'sweep_vmp': '20.935', 'sweep_pmax': '67.823', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.47', 'ahc_daily': '3.7', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844324417 ms)
2025-02-17 14:28:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844384423 ms)
2025-02-17 14:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844384423 ms)
2025-02-17 14:28:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:28:25.400523+00:00', 'runtime': '844380.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.944', 'adc_vl_f': '13.501', 'adc_ic_f': '5.266', 'adc_il_f': '0.336', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.501', 'vb_ref': '14.237', 'ahc_r': '2633.8', 'ahc_t': '2633.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '71.144', 'sweep_vmp': '20.993', 'sweep_pmax': '70.118', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.51', 'ahc_daily': '3.8', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844384424 ms)
2025-02-17 14:28:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:27:25.400189+00:00', 'runtime': '844320.001', 'adc_vb_f': '13.492', 'adc_va_f': '20.944', 'adc_vl_f': '13.501', 'adc_ic_f': '5.225', 'adc_il_f': '0.329', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.477', 'vb_ref': '14.237', 'ahc_r': '2633.8', 'ahc_t': '2633.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '70.541', 'sweep_vmp': '20.993', 'sweep_pmax': '70.118', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.498', 'ahc_daily': '3.8', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844384425 ms)
2025-02-17 14:28:35.705 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844393524 ms)
2025-02-17 14:28:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844393525 ms)
2025-02-17 14:28:35.707 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:28:25.405064+00:00'} (T+844393526 ms)
2025-02-17 14:29:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844444412 ms)
2025-02-17 14:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844444412 ms)
2025-02-17 14:29:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:29:25.400172+00:00', 'runtime': '844440.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.947', 'adc_vl_f': '13.522', 'adc_ic_f': '5.283', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.51', 'vb_ref': '14.237', 'ahc_r': '2633.9', 'ahc_t': '2633.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '71.371', 'sweep_vmp': '20.993', 'sweep_pmax': '70.118', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.51', 'ahc_daily': '3.9', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844444413 ms)
2025-02-17 14:29:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:28:25.400523+00:00', 'runtime': '844380.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.944', 'adc_vl_f': '13.501', 'adc_ic_f': '5.266', 'adc_il_f': '0.336', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.501', 'vb_ref': '14.237', 'ahc_r': '2633.8', 'ahc_t': '2633.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '71.144', 'sweep_vmp': '20.993', 'sweep_pmax': '70.118', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.51', 'ahc_daily': '3.8', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844444414 ms)
2025-02-17 14:29:48.618 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+844466437 ms)
2025-02-17 14:29:48.618 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+844466438 ms)
2025-02-17 14:29:48.619 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36804)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844466438 ms)
2025-02-17 14:30:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844504417 ms)
2025-02-17 14:30:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844504418 ms)
2025-02-17 14:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:30:25.400170+00:00', 'runtime': '844500.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.85', 'adc_vl_f': '13.535', 'adc_ic_f': '5.283', 'adc_il_f': '0.312', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.507', 'vb_ref': '14.237', 'ahc_r': '2634.0', 'ahc_t': '2634.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '71.371', 'sweep_vmp': '20.883', 'sweep_pmax': '71.084', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.51', 'ahc_daily': '4.0', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844504419 ms)
2025-02-17 14:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:29:25.400172+00:00', 'runtime': '844440.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.947', 'adc_vl_f': '13.522', 'adc_ic_f': '5.283', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.51', 'vb_ref': '14.237', 'ahc_r': '2633.9', 'ahc_t': '2633.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '71.371', 'sweep_vmp': '20.993', 'sweep_pmax': '70.118', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.51', 'ahc_daily': '3.9', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844504419 ms)
2025-02-17 14:31:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844564408 ms)
2025-02-17 14:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844564408 ms)
2025-02-17 14:31:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:31:25.400244+00:00', 'runtime': '844560.001', 'adc_vb_f': '13.525', 'adc_va_f': '20.85', 'adc_vl_f': '13.538', 'adc_ic_f': '5.349', 'adc_il_f': '0.331', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.513', 'vb_ref': '14.237', 'ahc_r': '2634.1', 'ahc_t': '2634.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '72.277', 'sweep_vmp': '20.883', 'sweep_pmax': '71.084', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.528', 'ahc_daily': '4.1', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844564409 ms)
2025-02-17 14:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:30:25.400170+00:00', 'runtime': '844500.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.85', 'adc_vl_f': '13.535', 'adc_ic_f': '5.283', 'adc_il_f': '0.312', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.507', 'vb_ref': '14.237', 'ahc_r': '2634.0', 'ahc_t': '2634.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '71.371', 'sweep_vmp': '20.883', 'sweep_pmax': '71.084', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.51', 'ahc_daily': '4.0', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844564410 ms)
2025-02-17 14:31:51.863 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+844589682 ms)
2025-02-17 14:31:51.863 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+844589683 ms)
2025-02-17 14:31:51.864 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36936)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844589683 ms)
2025-02-17 14:32:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844624413 ms)
2025-02-17 14:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844624414 ms)
2025-02-17 14:32:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:32:25.400187+00:00', 'runtime': '844620.001', 'adc_vb_f': '13.541', 'adc_va_f': '20.85', 'adc_vl_f': '13.538', 'adc_ic_f': '5.392', 'adc_il_f': '0.374', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.531', 'vb_ref': '14.237', 'ahc_r': '2634.2', 'ahc_t': '2634.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '73.017', 'sweep_vmp': '20.883', 'sweep_pmax': '71.084', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.547', 'ahc_daily': '4.2', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844624416 ms)
2025-02-17 14:32:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:31:25.400244+00:00', 'runtime': '844560.001', 'adc_vb_f': '13.525', 'adc_va_f': '20.85', 'adc_vl_f': '13.538', 'adc_ic_f': '5.349', 'adc_il_f': '0.331', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.513', 'vb_ref': '14.237', 'ahc_r': '2634.1', 'ahc_t': '2634.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '72.277', 'sweep_vmp': '20.883', 'sweep_pmax': '71.084', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.528', 'ahc_daily': '4.1', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844624416 ms)
2025-02-17 14:32:37.920 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+844635740 ms)
2025-02-17 14:32:37.921 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+844635740 ms)
2025-02-17 14:32:37.921 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844635740 ms)
2025-02-17 14:32:37.922 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+844635741 ms)
2025-02-17 14:32:37.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+844635741 ms)
2025-02-17 14:32:37.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844635741 ms)
2025-02-17 14:32:37.923 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+844635742 ms)
2025-02-17 14:32:37.923 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+844635742 ms)
2025-02-17 14:32:37.923 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844635742 ms)
2025-02-17 14:32:37.924 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+844635743 ms)
2025-02-17 14:32:37.924 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+844635743 ms)
2025-02-17 14:32:37.924 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844635744 ms)
2025-02-17 14:32:37.925 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+844635744 ms)
2025-02-17 14:32:37.925 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+844635744 ms)
2025-02-17 14:32:37.934 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844635753 ms)
2025-02-17 14:33:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844684403 ms)
2025-02-17 14:33:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844684404 ms)
2025-02-17 14:33:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:33:25.400177+00:00', 'runtime': '844680.001', 'adc_vb_f': '13.547', 'adc_va_f': '20.85', 'adc_vl_f': '13.538', 'adc_ic_f': '5.45', 'adc_il_f': '0.36', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.544', 'vb_ref': '14.237', 'ahc_r': '2634.3', 'ahc_t': '2634.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '73.892', 'sweep_vmp': '20.889', 'sweep_pmax': '73.651', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.547', 'ahc_daily': '4.3', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844684406 ms)
2025-02-17 14:33:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:32:25.400187+00:00', 'runtime': '844620.001', 'adc_vb_f': '13.541', 'adc_va_f': '20.85', 'adc_vl_f': '13.538', 'adc_ic_f': '5.392', 'adc_il_f': '0.374', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.531', 'vb_ref': '14.237', 'ahc_r': '2634.2', 'ahc_t': '2634.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '73.017', 'sweep_vmp': '20.883', 'sweep_pmax': '71.084', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.547', 'ahc_daily': '4.2', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844684406 ms)
2025-02-17 14:33:53.094 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+844710913 ms)
2025-02-17 14:33:53.094 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+844710913 ms)
2025-02-17 14:33:53.094 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37062)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844710914 ms)
2025-02-17 14:34:15.165 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+844732985 ms)
2025-02-17 14:34:15.166 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+844732985 ms)
2025-02-17 14:34:15.166 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37086)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844732985 ms)
2025-02-17 14:34:21.173 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+844738993 ms)
2025-02-17 14:34:21.174 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+844738993 ms)
2025-02-17 14:34:21.174 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+844738993 ms)
2025-02-17 14:34:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844744414 ms)
2025-02-17 14:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844744414 ms)
2025-02-17 14:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:34:25.400198+00:00', 'runtime': '844740.001', 'adc_vb_f': '13.562', 'adc_va_f': '20.85', 'adc_vl_f': '13.559', 'adc_ic_f': '5.484', 'adc_il_f': '0.362', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.556', 'vb_ref': '14.237', 'ahc_r': '2634.4', 'ahc_t': '2634.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '74.345', 'sweep_vmp': '20.889', 'sweep_pmax': '73.651', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.574', 'ahc_daily': '4.4', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844744417 ms)
2025-02-17 14:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:33:25.400177+00:00', 'runtime': '844680.001', 'adc_vb_f': '13.547', 'adc_va_f': '20.85', 'adc_vl_f': '13.538', 'adc_ic_f': '5.45', 'adc_il_f': '0.36', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.544', 'vb_ref': '14.237', 'ahc_r': '2634.3', 'ahc_t': '2634.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '73.892', 'sweep_vmp': '20.889', 'sweep_pmax': '73.651', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.547', 'ahc_daily': '4.3', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844744417 ms)
2025-02-17 14:34:35.726 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844753546 ms)
2025-02-17 14:34:35.727 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844753546 ms)
2025-02-17 14:34:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:34:25.413050+00:00'} (T+844753547 ms)
2025-02-17 14:35:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844804417 ms)
2025-02-17 14:35:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844804418 ms)
2025-02-17 14:35:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:35:25.400158+00:00', 'runtime': '844800.001', 'adc_vb_f': '13.586', 'adc_va_f': '20.85', 'adc_vl_f': '13.577', 'adc_ic_f': '5.532', 'adc_il_f': '0.362', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.58', 'vb_ref': '14.237', 'ahc_r': '2634.5', 'ahc_t': '2634.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.161', 'sweep_vmp': '20.889', 'sweep_pmax': '73.651', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.586', 'ahc_daily': '4.5', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844804420 ms)
2025-02-17 14:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:34:25.400198+00:00', 'runtime': '844740.001', 'adc_vb_f': '13.562', 'adc_va_f': '20.85', 'adc_vl_f': '13.559', 'adc_ic_f': '5.484', 'adc_il_f': '0.362', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.556', 'vb_ref': '14.237', 'ahc_r': '2634.4', 'ahc_t': '2634.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.7', 'ahl_t': '1590.7', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '74.345', 'sweep_vmp': '20.889', 'sweep_pmax': '73.651', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.574', 'ahc_daily': '4.4', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844804420 ms)
2025-02-17 14:35:35.750 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844813570 ms)
2025-02-17 14:35:35.751 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844813570 ms)
2025-02-17 14:35:35.752 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:35:25.404166+00:00'} (T+844813571 ms)
2025-02-17 14:36:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844864406 ms)
2025-02-17 14:36:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844864406 ms)
2025-02-17 14:36:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:36:25.400196+00:00', 'runtime': '844860.001', 'adc_vb_f': '13.586', 'adc_va_f': '20.853', 'adc_vl_f': '13.596', 'adc_ic_f': '5.559', 'adc_il_f': '0.348', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.583', 'vb_ref': '14.237', 'ahc_r': '2634.6', 'ahc_t': '2634.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.478', 'sweep_vmp': '20.859', 'sweep_pmax': '75.13', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.586', 'ahc_daily': '4.6', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844864407 ms)
2025-02-17 14:36:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:35:25.400158+00:00', 'runtime': '844800.001', 'adc_vb_f': '13.586', 'adc_va_f': '20.85', 'adc_vl_f': '13.577', 'adc_ic_f': '5.532', 'adc_il_f': '0.362', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.58', 'vb_ref': '14.237', 'ahc_r': '2634.5', 'ahc_t': '2634.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.161', 'sweep_vmp': '20.889', 'sweep_pmax': '73.651', 'sweep_voc': '23.352', 'vb_min_daily': '12.164', 'vb_max_daily': '13.586', 'ahc_daily': '4.5', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844864408 ms)
2025-02-17 14:36:35.684 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844873503 ms)
2025-02-17 14:36:35.685 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844873504 ms)
2025-02-17 14:36:35.686 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:36:25.400138+00:00'} (T+844873505 ms)
2025-02-17 14:37:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844924414 ms)
2025-02-17 14:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844924415 ms)
2025-02-17 14:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:37:25.400052+00:00', 'runtime': '844920.001', 'adc_vb_f': '13.589', 'adc_va_f': '20.853', 'adc_vl_f': '13.614', 'adc_ic_f': '5.571', 'adc_il_f': '0.324', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.586', 'vb_ref': '14.237', 'ahc_r': '2634.7', 'ahc_t': '2634.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.644', 'sweep_vmp': '20.859', 'sweep_pmax': '75.13', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.596', 'ahc_daily': '4.7', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844924416 ms)
2025-02-17 14:37:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:36:25.400196+00:00', 'runtime': '844860.001', 'adc_vb_f': '13.586', 'adc_va_f': '20.853', 'adc_vl_f': '13.596', 'adc_ic_f': '5.559', 'adc_il_f': '0.348', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.583', 'vb_ref': '14.237', 'ahc_r': '2634.6', 'ahc_t': '2634.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.478', 'sweep_vmp': '20.859', 'sweep_pmax': '75.13', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.586', 'ahc_daily': '4.6', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844924416 ms)
2025-02-17 14:37:35.886 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844933705 ms)
2025-02-17 14:37:35.886 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844933705 ms)
2025-02-17 14:37:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:37:25.400102+00:00'} (T+844933706 ms)
2025-02-17 14:38:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+844984419 ms)
2025-02-17 14:38:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+844984419 ms)
2025-02-17 14:38:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:38:25.400168+00:00', 'runtime': '844980.001', 'adc_vb_f': '13.593', 'adc_va_f': '20.85', 'adc_vl_f': '13.614', 'adc_ic_f': '5.539', 'adc_il_f': '0.319', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.589', 'vb_ref': '14.237', 'ahc_r': '2634.8', 'ahc_t': '2634.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.266', 'sweep_vmp': '20.859', 'sweep_pmax': '75.13', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.602', 'ahc_daily': '4.8', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844984423 ms)
2025-02-17 14:38:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:37:25.400052+00:00', 'runtime': '844920.001', 'adc_vb_f': '13.589', 'adc_va_f': '20.853', 'adc_vl_f': '13.614', 'adc_ic_f': '5.571', 'adc_il_f': '0.324', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.586', 'vb_ref': '14.237', 'ahc_r': '2634.7', 'ahc_t': '2634.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.644', 'sweep_vmp': '20.859', 'sweep_pmax': '75.13', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.596', 'ahc_daily': '4.7', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+844984424 ms)
2025-02-17 14:38:35.732 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844993551 ms)
2025-02-17 14:38:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+844993552 ms)
2025-02-17 14:38:35.733 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:38:25.405138+00:00'} (T+844993553 ms)
2025-02-17 14:39:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845044410 ms)
2025-02-17 14:39:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845044410 ms)
2025-02-17 14:39:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:39:25.402663+00:00', 'runtime': '845040.003', 'adc_vb_f': '13.614', 'adc_va_f': '20.755', 'adc_vl_f': '13.614', 'adc_ic_f': '5.564', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.602', 'vb_ref': '14.237', 'ahc_r': '2634.8', 'ahc_t': '2634.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.704', 'sweep_vmp': '20.792', 'sweep_pmax': '75.704', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.62', 'ahc_daily': '4.8', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845044411 ms)
2025-02-17 14:39:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:38:25.400168+00:00', 'runtime': '844980.001', 'adc_vb_f': '13.593', 'adc_va_f': '20.85', 'adc_vl_f': '13.614', 'adc_ic_f': '5.539', 'adc_il_f': '0.319', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.589', 'vb_ref': '14.237', 'ahc_r': '2634.8', 'ahc_t': '2634.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.266', 'sweep_vmp': '20.859', 'sweep_pmax': '75.13', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '13.602', 'ahc_daily': '4.8', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845044415 ms)
2025-02-17 14:39:35.900 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845053719 ms)
2025-02-17 14:39:35.900 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845053720 ms)
2025-02-17 14:39:35.904 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:39:25.400078+00:00'} (T+845053723 ms)
2025-02-17 14:40:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845104417 ms)
2025-02-17 14:40:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845104417 ms)
2025-02-17 14:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:40:25.400181+00:00', 'runtime': '845100.001', 'adc_vb_f': '13.623', 'adc_va_f': '20.755', 'adc_vl_f': '13.614', 'adc_ic_f': '5.665', 'adc_il_f': '0.345', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.62', 'vb_ref': '14.237', 'ahc_r': '2634.9', 'ahc_t': '2634.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.169', 'sweep_vmp': '20.792', 'sweep_pmax': '75.704', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.626', 'ahc_daily': '4.9', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845104421 ms)
2025-02-17 14:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:39:25.402663+00:00', 'runtime': '845040.003', 'adc_vb_f': '13.614', 'adc_va_f': '20.755', 'adc_vl_f': '13.614', 'adc_ic_f': '5.564', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.602', 'vb_ref': '14.237', 'ahc_r': '2634.8', 'ahc_t': '2634.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.704', 'sweep_vmp': '20.792', 'sweep_pmax': '75.704', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.62', 'ahc_daily': '4.8', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845104422 ms)
2025-02-17 14:40:35.712 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845113531 ms)
2025-02-17 14:40:35.713 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845113532 ms)
2025-02-17 14:40:35.714 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:40:25.401149+00:00'} (T+845113533 ms)
2025-02-17 14:41:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845164407 ms)
2025-02-17 14:41:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845164407 ms)
2025-02-17 14:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:41:25.402736+00:00', 'runtime': '845160.003', 'adc_vb_f': '13.623', 'adc_va_f': '20.755', 'adc_vl_f': '13.614', 'adc_ic_f': '5.609', 'adc_il_f': '0.324', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.623', 'vb_ref': '14.237', 'ahc_r': '2635.0', 'ahc_t': '2635.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '76.384', 'sweep_vmp': '20.792', 'sweep_pmax': '75.704', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.632', 'ahc_daily': '5.0', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845164411 ms)
2025-02-17 14:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:40:25.400181+00:00', 'runtime': '845100.001', 'adc_vb_f': '13.623', 'adc_va_f': '20.755', 'adc_vl_f': '13.614', 'adc_ic_f': '5.665', 'adc_il_f': '0.345', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.62', 'vb_ref': '14.237', 'ahc_r': '2634.9', 'ahc_t': '2634.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.169', 'sweep_vmp': '20.792', 'sweep_pmax': '75.704', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.626', 'ahc_daily': '4.9', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845164411 ms)
2025-02-17 14:42:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845224420 ms)
2025-02-17 14:42:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845224420 ms)
2025-02-17 14:42:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:42:25.401004+00:00', 'runtime': '845220.001', 'adc_vb_f': '13.617', 'adc_va_f': '20.807', 'adc_vl_f': '13.614', 'adc_ic_f': '5.496', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.617', 'vb_ref': '14.237', 'ahc_r': '2635.1', 'ahc_t': '2635.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '74.889', 'sweep_vmp': '20.801', 'sweep_pmax': '75.478', 'sweep_voc': '23.291', 'vb_min_daily': '12.164', 'vb_max_daily': '13.632', 'ahc_daily': '5.1', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845224425 ms)
2025-02-17 14:42:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:41:25.402736+00:00', 'runtime': '845160.003', 'adc_vb_f': '13.623', 'adc_va_f': '20.755', 'adc_vl_f': '13.614', 'adc_ic_f': '5.609', 'adc_il_f': '0.324', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.623', 'vb_ref': '14.237', 'ahc_r': '2635.0', 'ahc_t': '2635.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '76.384', 'sweep_vmp': '20.792', 'sweep_pmax': '75.704', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.632', 'ahc_daily': '5.0', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845224425 ms)
2025-02-17 14:43:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845284408 ms)
2025-02-17 14:43:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845284408 ms)
2025-02-17 14:43:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:43:25.402767+00:00', 'runtime': '845280.003', 'adc_vb_f': '13.623', 'adc_va_f': '20.804', 'adc_vl_f': '13.614', 'adc_ic_f': '5.551', 'adc_il_f': '0.341', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.596', 'vb_ref': '14.237', 'ahc_r': '2635.2', 'ahc_t': '2635.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.583', 'sweep_vmp': '20.801', 'sweep_pmax': '75.478', 'sweep_voc': '23.291', 'vb_min_daily': '12.164', 'vb_max_daily': '13.632', 'ahc_daily': '5.2', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845284412 ms)
2025-02-17 14:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:42:25.401004+00:00', 'runtime': '845220.001', 'adc_vb_f': '13.617', 'adc_va_f': '20.807', 'adc_vl_f': '13.614', 'adc_ic_f': '5.496', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.617', 'vb_ref': '14.237', 'ahc_r': '2635.1', 'ahc_t': '2635.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '74.889', 'sweep_vmp': '20.801', 'sweep_pmax': '75.478', 'sweep_voc': '23.291', 'vb_min_daily': '12.164', 'vb_max_daily': '13.632', 'ahc_daily': '5.1', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845284413 ms)
2025-02-17 14:43:35.827 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845293647 ms)
2025-02-17 14:43:35.828 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845293647 ms)
2025-02-17 14:43:35.832 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:43:25.400115+00:00'} (T+845293651 ms)
2025-02-17 14:44:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845344418 ms)
2025-02-17 14:44:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845344418 ms)
2025-02-17 14:44:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:44:25.400004+00:00', 'runtime': '845340.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.804', 'adc_vl_f': '13.69', 'adc_ic_f': '5.75', 'adc_il_f': '0.353', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.654', 'vb_ref': '14.237', 'ahc_r': '2635.3', 'ahc_t': '2635.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '78.482', 'sweep_vmp': '20.801', 'sweep_pmax': '75.478', 'sweep_voc': '23.291', 'vb_min_daily': '12.164', 'vb_max_daily': '13.663', 'ahc_daily': '5.3', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845344419 ms)
2025-02-17 14:44:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:43:25.402767+00:00', 'runtime': '845280.003', 'adc_vb_f': '13.623', 'adc_va_f': '20.804', 'adc_vl_f': '13.614', 'adc_ic_f': '5.551', 'adc_il_f': '0.341', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.596', 'vb_ref': '14.237', 'ahc_r': '2635.2', 'ahc_t': '2635.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '75.583', 'sweep_vmp': '20.801', 'sweep_pmax': '75.478', 'sweep_voc': '23.291', 'vb_min_daily': '12.164', 'vb_max_daily': '13.632', 'ahc_daily': '5.2', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845344420 ms)
2025-02-17 14:44:35.629 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845353449 ms)
2025-02-17 14:44:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845353449 ms)
2025-02-17 14:44:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:44:25.405133+00:00'} (T+845353450 ms)
2025-02-17 14:44:42.013 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+845359833 ms)
2025-02-17 14:44:42.014 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+845359833 ms)
2025-02-17 14:44:42.014 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37762)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+845359833 ms)
2025-02-17 14:45:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845404408 ms)
2025-02-17 14:45:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845404408 ms)
2025-02-17 14:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:45:25.402841+00:00', 'runtime': '845400.003', 'adc_vb_f': '13.663', 'adc_va_f': '20.755', 'adc_vl_f': '13.657', 'adc_ic_f': '5.643', 'adc_il_f': '0.365', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.657', 'vb_ref': '14.237', 'ahc_r': '2635.4', 'ahc_t': '2635.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.093', 'sweep_vmp': '20.795', 'sweep_pmax': '77.425', 'sweep_voc': '23.276', 'vb_min_daily': '12.164', 'vb_max_daily': '13.663', 'ahc_daily': '5.4', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845404410 ms)
2025-02-17 14:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:44:25.400004+00:00', 'runtime': '845340.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.804', 'adc_vl_f': '13.69', 'adc_ic_f': '5.75', 'adc_il_f': '0.353', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.654', 'vb_ref': '14.237', 'ahc_r': '2635.3', 'ahc_t': '2635.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '78.482', 'sweep_vmp': '20.801', 'sweep_pmax': '75.478', 'sweep_voc': '23.291', 'vb_min_daily': '12.164', 'vb_max_daily': '13.663', 'ahc_daily': '5.3', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845404411 ms)
2025-02-17 14:46:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845464413 ms)
2025-02-17 14:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845464413 ms)
2025-02-17 14:46:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:46:25.400624+00:00', 'runtime': '845460.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.755', 'adc_vl_f': '13.69', 'adc_ic_f': '5.677', 'adc_il_f': '0.36', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.666', 'vb_ref': '14.237', 'ahc_r': '2635.5', 'ahc_t': '2635.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.561', 'sweep_vmp': '20.795', 'sweep_pmax': '77.425', 'sweep_voc': '23.276', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.5', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845464415 ms)
2025-02-17 14:46:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:45:25.402841+00:00', 'runtime': '845400.003', 'adc_vb_f': '13.663', 'adc_va_f': '20.755', 'adc_vl_f': '13.657', 'adc_ic_f': '5.643', 'adc_il_f': '0.365', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.657', 'vb_ref': '14.237', 'ahc_r': '2635.4', 'ahc_t': '2635.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.093', 'sweep_vmp': '20.795', 'sweep_pmax': '77.425', 'sweep_voc': '23.276', 'vb_min_daily': '12.164', 'vb_max_daily': '13.663', 'ahc_daily': '5.4', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845464415 ms)
2025-02-17 14:47:18.276 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+845516095 ms)
2025-02-17 14:47:18.276 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+845516095 ms)
2025-02-17 14:47:18.277 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37938)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+845516096 ms)
2025-02-17 14:47:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845524410 ms)
2025-02-17 14:47:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845524411 ms)
2025-02-17 14:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:47:25.400615+00:00', 'runtime': '845520.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.755', 'adc_vl_f': '13.69', 'adc_ic_f': '5.648', 'adc_il_f': '0.35', 't_hs': '11', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.237', 'ahc_r': '2635.6', 'ahc_t': '2635.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.093', 'sweep_vmp': '20.795', 'sweep_pmax': '77.425', 'sweep_voc': '23.276', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.6', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845524412 ms)
2025-02-17 14:47:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:46:25.400624+00:00', 'runtime': '845460.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.755', 'adc_vl_f': '13.69', 'adc_ic_f': '5.677', 'adc_il_f': '0.36', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.666', 'vb_ref': '14.237', 'ahc_r': '2635.5', 'ahc_t': '2635.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.561', 'sweep_vmp': '20.795', 'sweep_pmax': '77.425', 'sweep_voc': '23.276', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.5', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845524412 ms)
2025-02-17 14:48:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845584413 ms)
2025-02-17 14:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845584414 ms)
2025-02-17 14:48:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:48:25.400174+00:00', 'runtime': '845580.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.807', 'adc_vl_f': '13.69', 'adc_ic_f': '5.655', 'adc_il_f': '0.372', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.207', 'ahc_r': '2635.7', 'ahc_t': '2635.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.29', 'sweep_vmp': '20.801', 'sweep_pmax': '77.108', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.7', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845584416 ms)
2025-02-17 14:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:47:25.400615+00:00', 'runtime': '845520.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.755', 'adc_vl_f': '13.69', 'adc_ic_f': '5.648', 'adc_il_f': '0.35', 't_hs': '11', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.237', 'ahc_r': '2635.6', 'ahc_t': '2635.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.093', 'sweep_vmp': '20.795', 'sweep_pmax': '77.425', 'sweep_voc': '23.276', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.6', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845584416 ms)
2025-02-17 14:49:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845644403 ms)
2025-02-17 14:49:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845644403 ms)
2025-02-17 14:49:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:49:25.400607+00:00', 'runtime': '845640.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.804', 'adc_vl_f': '13.69', 'adc_ic_f': '5.583', 'adc_il_f': '0.302', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.207', 'ahc_r': '2635.8', 'ahc_t': '2635.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '76.233', 'sweep_vmp': '20.801', 'sweep_pmax': '77.108', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.8', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845644404 ms)
2025-02-17 14:49:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:48:25.400174+00:00', 'runtime': '845580.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.807', 'adc_vl_f': '13.69', 'adc_ic_f': '5.655', 'adc_il_f': '0.372', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.207', 'ahc_r': '2635.7', 'ahc_t': '2635.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.29', 'sweep_vmp': '20.801', 'sweep_pmax': '77.108', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.7', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845644405 ms)
2025-02-17 14:50:06.486 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+845684305 ms)
2025-02-17 14:50:06.486 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+845684305 ms)
2025-02-17 14:50:06.486 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+845684306 ms)
2025-02-17 14:50:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845704413 ms)
2025-02-17 14:50:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845704413 ms)
2025-02-17 14:50:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:50:25.400602+00:00', 'runtime': '845700.001', 'adc_vb_f': '13.687', 'adc_va_f': '20.807', 'adc_vl_f': '13.69', 'adc_ic_f': '5.641', 'adc_il_f': '0.341', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.675', 'vb_ref': '14.207', 'ahc_r': '2635.9', 'ahc_t': '2635.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.32', 'sweep_vmp': '20.801', 'sweep_pmax': '77.108', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.69', 'ahc_daily': '5.9', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845704414 ms)
2025-02-17 14:50:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:49:25.400607+00:00', 'runtime': '845640.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.804', 'adc_vl_f': '13.69', 'adc_ic_f': '5.583', 'adc_il_f': '0.302', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.207', 'ahc_r': '2635.8', 'ahc_t': '2635.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '76.233', 'sweep_vmp': '20.801', 'sweep_pmax': '77.108', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.684', 'ahc_daily': '5.8', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845704414 ms)
2025-02-17 14:51:11.644 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+845749464 ms)
2025-02-17 14:51:11.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+845749464 ms)
2025-02-17 14:51:11.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38182)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+845749464 ms)
2025-02-17 14:51:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845764418 ms)
2025-02-17 14:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845764419 ms)
2025-02-17 14:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:51:25.402729+00:00', 'runtime': '845760.003', 'adc_vb_f': '13.721', 'adc_va_f': '20.85', 'adc_vl_f': '13.73', 'adc_ic_f': '5.752', 'adc_il_f': '0.319', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.693', 'vb_ref': '14.207', 'ahc_r': '2636.0', 'ahc_t': '2636.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '78.89', 'sweep_vmp': '20.865', 'sweep_pmax': '77.456', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.724', 'ahc_daily': '6.0', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845764420 ms)
2025-02-17 14:51:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:50:25.400602+00:00', 'runtime': '845700.001', 'adc_vb_f': '13.687', 'adc_va_f': '20.807', 'adc_vl_f': '13.69', 'adc_ic_f': '5.641', 'adc_il_f': '0.341', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.675', 'vb_ref': '14.207', 'ahc_r': '2635.9', 'ahc_t': '2635.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '77.32', 'sweep_vmp': '20.801', 'sweep_pmax': '77.108', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '13.69', 'ahc_daily': '5.9', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845764420 ms)
2025-02-17 14:52:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845824409 ms)
2025-02-17 14:52:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845824410 ms)
2025-02-17 14:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:52:25.400585+00:00', 'runtime': '845820.001', 'adc_vb_f': '13.739', 'adc_va_f': '20.85', 'adc_vl_f': '13.766', 'adc_ic_f': '5.793', 'adc_il_f': '0.319', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.73', 'vb_ref': '14.207', 'ahc_r': '2636.1', 'ahc_t': '2636.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '79.615', 'sweep_vmp': '20.865', 'sweep_pmax': '77.456', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.739', 'ahc_daily': '6.1', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845824411 ms)
2025-02-17 14:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:51:25.402729+00:00', 'runtime': '845760.003', 'adc_vb_f': '13.721', 'adc_va_f': '20.85', 'adc_vl_f': '13.73', 'adc_ic_f': '5.752', 'adc_il_f': '0.319', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.693', 'vb_ref': '14.207', 'ahc_r': '2636.0', 'ahc_t': '2636.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '78.89', 'sweep_vmp': '20.865', 'sweep_pmax': '77.456', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.724', 'ahc_daily': '6.0', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845824411 ms)
2025-02-17 14:53:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845884417 ms)
2025-02-17 14:53:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845884417 ms)
2025-02-17 14:53:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:53:25.402770+00:00', 'runtime': '845880.003', 'adc_vb_f': '13.815', 'adc_va_f': '20.85', 'adc_vl_f': '13.806', 'adc_ic_f': '5.998', 'adc_il_f': '0.319', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.785', 'vb_ref': '14.207', 'ahc_r': '2636.2', 'ahc_t': '2636.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '82.831', 'sweep_vmp': '20.865', 'sweep_pmax': '77.456', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.815', 'ahc_daily': '6.2', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845884418 ms)
2025-02-17 14:53:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:52:25.400585+00:00', 'runtime': '845820.001', 'adc_vb_f': '13.739', 'adc_va_f': '20.85', 'adc_vl_f': '13.766', 'adc_ic_f': '5.793', 'adc_il_f': '0.319', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.73', 'vb_ref': '14.207', 'ahc_r': '2636.1', 'ahc_t': '2636.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '79.615', 'sweep_vmp': '20.865', 'sweep_pmax': '77.456', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.739', 'ahc_daily': '6.1', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845884419 ms)
2025-02-17 14:53:35.882 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845893702 ms)
2025-02-17 14:53:35.883 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+845893702 ms)
2025-02-17 14:53:35.884 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:53:25.400145+00:00'} (T+845893703 ms)
2025-02-17 14:54:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+845944424 ms)
2025-02-17 14:54:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+845944425 ms)
2025-02-17 14:54:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:54:25.400247+00:00', 'runtime': '845940.001', 'adc_vb_f': '13.824', 'adc_va_f': '20.755', 'adc_vl_f': '13.837', 'adc_ic_f': '6.015', 'adc_il_f': '0.331', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.806', 'vb_ref': '14.207', 'ahc_r': '2636.3', 'ahc_t': '2636.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '83.133', 'sweep_vmp': '20.786', 'sweep_pmax': '82.785', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.837', 'ahc_daily': '6.3', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845944426 ms)
2025-02-17 14:54:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:53:25.402770+00:00', 'runtime': '845880.003', 'adc_vb_f': '13.815', 'adc_va_f': '20.85', 'adc_vl_f': '13.806', 'adc_ic_f': '5.998', 'adc_il_f': '0.319', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.785', 'vb_ref': '14.207', 'ahc_r': '2636.2', 'ahc_t': '2636.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20121', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '82.831', 'sweep_vmp': '20.865', 'sweep_pmax': '77.456', 'sweep_voc': '23.343', 'vb_min_daily': '12.164', 'vb_max_daily': '13.815', 'ahc_daily': '6.2', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+845944426 ms)
2025-02-17 14:55:12.015 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+845989834 ms)
2025-02-17 14:55:12.015 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+845989834 ms)
2025-02-17 14:55:12.015 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38438)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+845989834 ms)
2025-02-17 14:55:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846004416 ms)
2025-02-17 14:55:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846004416 ms)
2025-02-17 14:55:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:55:25.399997+00:00', 'runtime': '846000.001', 'adc_vb_f': '13.852', 'adc_va_f': '20.755', 'adc_vl_f': '13.843', 'adc_ic_f': '6.044', 'adc_il_f': '0.345', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.843', 'vb_ref': '14.207', 'ahc_r': '2636.4', 'ahc_t': '2636.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '83.586', 'sweep_vmp': '20.786', 'sweep_pmax': '82.785', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.861', 'ahc_daily': '6.4', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846004417 ms)
2025-02-17 14:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:54:25.400247+00:00', 'runtime': '845940.001', 'adc_vb_f': '13.824', 'adc_va_f': '20.755', 'adc_vl_f': '13.837', 'adc_ic_f': '6.015', 'adc_il_f': '0.331', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.806', 'vb_ref': '14.207', 'ahc_r': '2636.3', 'ahc_t': '2636.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '83.133', 'sweep_vmp': '20.786', 'sweep_pmax': '82.785', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.837', 'ahc_daily': '6.3', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846004418 ms)
2025-02-17 14:56:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846064407 ms)
2025-02-17 14:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846064408 ms)
2025-02-17 14:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:56:25.400153+00:00', 'runtime': '846060.001', 'adc_vb_f': '13.892', 'adc_va_f': '20.755', 'adc_vl_f': '13.882', 'adc_ic_f': '6.122', 'adc_il_f': '0.341', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.873', 'vb_ref': '14.207', 'ahc_r': '2636.5', 'ahc_t': '2636.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '84.899', 'sweep_vmp': '20.786', 'sweep_pmax': '82.785', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.892', 'ahc_daily': '6.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846064409 ms)
2025-02-17 14:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:55:25.399997+00:00', 'runtime': '846000.001', 'adc_vb_f': '13.852', 'adc_va_f': '20.755', 'adc_vl_f': '13.843', 'adc_ic_f': '6.044', 'adc_il_f': '0.345', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.843', 'vb_ref': '14.207', 'ahc_r': '2636.4', 'ahc_t': '2636.4', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.8', 'ahl_t': '1590.8', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '83.586', 'sweep_vmp': '20.786', 'sweep_pmax': '82.785', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.861', 'ahc_daily': '6.4', 'ahl_daily': '6.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846064409 ms)
2025-02-17 14:57:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846124415 ms)
2025-02-17 14:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846124415 ms)
2025-02-17 14:57:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:57:25.402762+00:00', 'runtime': '846120.003', 'adc_vb_f': '13.922', 'adc_va_f': '20.755', 'adc_vl_f': '13.913', 'adc_ic_f': '6.165', 'adc_il_f': '0.36', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.895', 'vb_ref': '14.207', 'ahc_r': '2636.6', 'ahc_t': '2636.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '85.82', 'sweep_vmp': '20.798', 'sweep_pmax': '85.458', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.928', 'ahc_daily': '6.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846124416 ms)
2025-02-17 14:57:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:56:25.400153+00:00', 'runtime': '846060.001', 'adc_vb_f': '13.892', 'adc_va_f': '20.755', 'adc_vl_f': '13.882', 'adc_ic_f': '6.122', 'adc_il_f': '0.341', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.873', 'vb_ref': '14.207', 'ahc_r': '2636.5', 'ahc_t': '2636.5', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '84.899', 'sweep_vmp': '20.786', 'sweep_pmax': '82.785', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.892', 'ahc_daily': '6.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846124417 ms)
2025-02-17 14:57:35.696 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846133515 ms)
2025-02-17 14:57:35.696 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846133516 ms)
2025-02-17 14:57:35.697 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:57:25.400138+00:00'} (T+846133517 ms)
2025-02-17 14:58:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846184408 ms)
2025-02-17 14:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846184408 ms)
2025-02-17 14:58:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:58:25.402678+00:00', 'runtime': '846180.003', 'adc_vb_f': '13.928', 'adc_va_f': '20.755', 'adc_vl_f': '13.919', 'adc_ic_f': '6.177', 'adc_il_f': '0.362', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.925', 'vb_ref': '14.207', 'ahc_r': '2636.7', 'ahc_t': '2636.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '86.032', 'sweep_vmp': '20.798', 'sweep_pmax': '85.458', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.937', 'ahc_daily': '6.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846184409 ms)
2025-02-17 14:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:57:25.402762+00:00', 'runtime': '846120.003', 'adc_vb_f': '13.922', 'adc_va_f': '20.755', 'adc_vl_f': '13.913', 'adc_ic_f': '6.165', 'adc_il_f': '0.36', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.895', 'vb_ref': '14.207', 'ahc_r': '2636.6', 'ahc_t': '2636.6', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '85.82', 'sweep_vmp': '20.798', 'sweep_pmax': '85.458', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.928', 'ahc_daily': '6.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846184410 ms)
2025-02-17 14:58:35.736 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846193555 ms)
2025-02-17 14:58:35.737 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846193556 ms)
2025-02-17 14:58:35.738 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 14:58:25.400100+00:00'} (T+846193557 ms)
2025-02-17 14:59:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846244418 ms)
2025-02-17 14:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846244418 ms)
2025-02-17 14:59:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 14:59:25.400077+00:00', 'runtime': '846240.001', 'adc_vb_f': '13.968', 'adc_va_f': '20.755', 'adc_vl_f': '13.959', 'adc_ic_f': '6.211', 'adc_il_f': '0.338', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.947', 'vb_ref': '14.207', 'ahc_r': '2636.8', 'ahc_t': '2636.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '86.817', 'sweep_vmp': '20.798', 'sweep_pmax': '85.458', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.968', 'ahc_daily': '6.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846244419 ms)
2025-02-17 14:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:58:25.402678+00:00', 'runtime': '846180.003', 'adc_vb_f': '13.928', 'adc_va_f': '20.755', 'adc_vl_f': '13.919', 'adc_ic_f': '6.177', 'adc_il_f': '0.362', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.925', 'vb_ref': '14.207', 'ahc_r': '2636.7', 'ahc_t': '2636.7', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '86.032', 'sweep_vmp': '20.798', 'sweep_pmax': '85.458', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.937', 'ahc_daily': '6.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846244420 ms)
2025-02-17 15:00:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846304407 ms)
2025-02-17 15:00:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846304408 ms)
2025-02-17 15:00:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:00:25.400702+00:00', 'runtime': '846300.001', 'adc_vb_f': '13.995', 'adc_va_f': '20.807', 'adc_vl_f': '13.995', 'adc_ic_f': '6.288', 'adc_il_f': '0.297', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.953', 'vb_ref': '14.207', 'ahc_r': '2636.9', 'ahc_t': '2636.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '87.994', 'sweep_vmp': '20.816', 'sweep_pmax': '87.104', 'sweep_voc': '23.294', 'vb_min_daily': '12.164', 'vb_max_daily': '13.995', 'ahc_daily': '6.9', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846304409 ms)
2025-02-17 15:00:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 14:59:25.400077+00:00', 'runtime': '846240.001', 'adc_vb_f': '13.968', 'adc_va_f': '20.755', 'adc_vl_f': '13.959', 'adc_ic_f': '6.211', 'adc_il_f': '0.338', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.947', 'vb_ref': '14.207', 'ahc_r': '2636.8', 'ahc_t': '2636.8', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '86.817', 'sweep_vmp': '20.798', 'sweep_pmax': '85.458', 'sweep_voc': '23.376', 'vb_min_daily': '12.164', 'vb_max_daily': '13.968', 'ahc_daily': '6.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846304409 ms)
2025-02-17 15:01:23.548 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+846361368 ms)
2025-02-17 15:01:23.549 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+846361368 ms)
2025-02-17 15:01:23.549 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38838)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+846361368 ms)
2025-02-17 15:01:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846364416 ms)
2025-02-17 15:01:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846364417 ms)
2025-02-17 15:01:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:01:25.400164+00:00', 'runtime': '846360.001', 'adc_vb_f': '14.023', 'adc_va_f': '20.804', 'adc_vl_f': '14.029', 'adc_ic_f': '6.312', 'adc_il_f': '0.321', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.008', 'vb_ref': '14.207', 'ahc_r': '2637.0', 'ahc_t': '2637.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '88.402', 'sweep_vmp': '20.816', 'sweep_pmax': '87.104', 'sweep_voc': '23.294', 'vb_min_daily': '12.164', 'vb_max_daily': '14.041', 'ahc_daily': '7.0', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846364418 ms)
2025-02-17 15:01:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:00:25.400702+00:00', 'runtime': '846300.001', 'adc_vb_f': '13.995', 'adc_va_f': '20.807', 'adc_vl_f': '13.995', 'adc_ic_f': '6.288', 'adc_il_f': '0.297', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.953', 'vb_ref': '14.207', 'ahc_r': '2636.9', 'ahc_t': '2636.9', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '87.994', 'sweep_vmp': '20.816', 'sweep_pmax': '87.104', 'sweep_voc': '23.294', 'vb_min_daily': '12.164', 'vb_max_daily': '13.995', 'ahc_daily': '6.9', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846364418 ms)
2025-02-17 15:02:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846424408 ms)
2025-02-17 15:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846424408 ms)
2025-02-17 15:02:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:02:25.402870+00:00', 'runtime': '846420.003', 'adc_vb_f': '14.05', 'adc_va_f': '20.807', 'adc_vl_f': '14.035', 'adc_ic_f': '6.366', 'adc_il_f': '0.314', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.038', 'vb_ref': '14.207', 'ahc_r': '2637.1', 'ahc_t': '2637.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '89.429', 'sweep_vmp': '20.816', 'sweep_pmax': '87.104', 'sweep_voc': '23.294', 'vb_min_daily': '12.164', 'vb_max_daily': '14.062', 'ahc_daily': '7.1', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846424409 ms)
2025-02-17 15:02:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:01:25.400164+00:00', 'runtime': '846360.001', 'adc_vb_f': '14.023', 'adc_va_f': '20.804', 'adc_vl_f': '14.029', 'adc_ic_f': '6.312', 'adc_il_f': '0.321', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.008', 'vb_ref': '14.207', 'ahc_r': '2637.0', 'ahc_t': '2637.0', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '88.402', 'sweep_vmp': '20.816', 'sweep_pmax': '87.104', 'sweep_voc': '23.294', 'vb_min_daily': '12.164', 'vb_max_daily': '14.041', 'ahc_daily': '7.0', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846424410 ms)
2025-02-17 15:02:35.967 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846433786 ms)
2025-02-17 15:02:35.967 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846433786 ms)
2025-02-17 15:02:35.968 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:02:25.400196+00:00'} (T+846433787 ms)
2025-02-17 15:03:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846484415 ms)
2025-02-17 15:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846484416 ms)
2025-02-17 15:03:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:03:25.402805+00:00', 'runtime': '846480.003', 'adc_vb_f': '14.072', 'adc_va_f': '20.804', 'adc_vl_f': '14.059', 'adc_ic_f': '6.395', 'adc_il_f': '0.316', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.047', 'vb_ref': '14.207', 'ahc_r': '2637.2', 'ahc_t': '2637.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '89.837', 'sweep_vmp': '20.837', 'sweep_pmax': '89.323', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '14.081', 'ahc_daily': '7.2', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846484417 ms)
2025-02-17 15:03:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:02:25.402870+00:00', 'runtime': '846420.003', 'adc_vb_f': '14.05', 'adc_va_f': '20.807', 'adc_vl_f': '14.035', 'adc_ic_f': '6.366', 'adc_il_f': '0.314', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.038', 'vb_ref': '14.207', 'ahc_r': '2637.1', 'ahc_t': '2637.1', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '89.429', 'sweep_vmp': '20.816', 'sweep_pmax': '87.104', 'sweep_voc': '23.294', 'vb_min_daily': '12.164', 'vb_max_daily': '14.062', 'ahc_daily': '7.1', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846484417 ms)
2025-02-17 15:03:35.886 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846493705 ms)
2025-02-17 15:03:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846493706 ms)
2025-02-17 15:03:35.888 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:03:25.400222+00:00'} (T+846493707 ms)
2025-02-17 15:04:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846544424 ms)
2025-02-17 15:04:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846544425 ms)
2025-02-17 15:04:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:04:25.400186+00:00', 'runtime': '846540.001', 'adc_vb_f': '14.117', 'adc_va_f': '20.804', 'adc_vl_f': '14.111', 'adc_ic_f': '6.44', 'adc_il_f': '0.324', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.099', 'vb_ref': '14.207', 'ahc_r': '2637.3', 'ahc_t': '2637.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '90.954', 'sweep_vmp': '20.837', 'sweep_pmax': '89.323', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '14.117', 'ahc_daily': '7.3', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846544426 ms)
2025-02-17 15:04:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:03:25.402805+00:00', 'runtime': '846480.003', 'adc_vb_f': '14.072', 'adc_va_f': '20.804', 'adc_vl_f': '14.059', 'adc_ic_f': '6.395', 'adc_il_f': '0.316', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.047', 'vb_ref': '14.207', 'ahc_r': '2637.2', 'ahc_t': '2637.2', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '89.837', 'sweep_vmp': '20.837', 'sweep_pmax': '89.323', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '14.081', 'ahc_daily': '7.2', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846544427 ms)
2025-02-17 15:04:35.633 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846553453 ms)
2025-02-17 15:04:35.634 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846553453 ms)
2025-02-17 15:04:35.635 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:04:25.413136+00:00'} (T+846553454 ms)
2025-02-17 15:05:13.908 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+846591727 ms)
2025-02-17 15:05:13.908 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+846591728 ms)
2025-02-17 15:05:13.909 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39084)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+846591728 ms)
2025-02-17 15:05:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846604414 ms)
2025-02-17 15:05:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846604414 ms)
2025-02-17 15:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:05:25.402772+00:00', 'runtime': '846600.003', 'adc_vb_f': '14.127', 'adc_va_f': '20.807', 'adc_vl_f': '14.111', 'adc_ic_f': '6.469', 'adc_il_f': '0.345', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.117', 'vb_ref': '14.207', 'ahc_r': '2637.4', 'ahc_t': '2637.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '91.437', 'sweep_vmp': '20.837', 'sweep_pmax': '89.323', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '14.148', 'ahc_daily': '7.4', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846604415 ms)
2025-02-17 15:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:04:25.400186+00:00', 'runtime': '846540.001', 'adc_vb_f': '14.117', 'adc_va_f': '20.804', 'adc_vl_f': '14.111', 'adc_ic_f': '6.44', 'adc_il_f': '0.324', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.099', 'vb_ref': '14.207', 'ahc_r': '2637.3', 'ahc_t': '2637.3', 'kwhc': '34.1', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '90.954', 'sweep_vmp': '20.837', 'sweep_pmax': '89.323', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '14.117', 'ahc_daily': '7.3', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846604416 ms)
2025-02-17 15:05:35.773 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846613593 ms)
2025-02-17 15:05:35.774 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846613593 ms)
2025-02-17 15:05:35.776 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:05:25.400184+00:00'} (T+846613596 ms)
2025-02-17 15:06:11.073 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+846648893 ms)
2025-02-17 15:06:11.074 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+846648893 ms)
2025-02-17 15:06:11.074 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39156)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+846648893 ms)
2025-02-17 15:06:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846664408 ms)
2025-02-17 15:06:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846664409 ms)
2025-02-17 15:06:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:06:25.402639+00:00', 'runtime': '846660.003', 'adc_vb_f': '14.117', 'adc_va_f': '20.755', 'adc_vl_f': '14.111', 'adc_ic_f': '6.527', 'adc_il_f': '0.345', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.124', 'vb_ref': '14.207', 'ahc_r': '2637.5', 'ahc_t': '2637.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '92.071', 'sweep_vmp': '20.792', 'sweep_pmax': '92.041', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '14.163', 'ahc_daily': '7.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846664410 ms)
2025-02-17 15:06:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:05:25.402772+00:00', 'runtime': '846600.003', 'adc_vb_f': '14.127', 'adc_va_f': '20.807', 'adc_vl_f': '14.111', 'adc_ic_f': '6.469', 'adc_il_f': '0.345', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.117', 'vb_ref': '14.207', 'ahc_r': '2637.4', 'ahc_t': '2637.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '91.437', 'sweep_vmp': '20.837', 'sweep_pmax': '89.323', 'sweep_voc': '23.3', 'vb_min_daily': '12.164', 'vb_max_daily': '14.148', 'ahc_daily': '7.4', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846664410 ms)
2025-02-17 15:06:35.741 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846673560 ms)
2025-02-17 15:06:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846673561 ms)
2025-02-17 15:06:35.743 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:06:25.400042+00:00'} (T+846673562 ms)
2025-02-17 15:06:36.164 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+846673984 ms)
2025-02-17 15:06:36.165 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+846673984 ms)
2025-02-17 15:06:36.165 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39184)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+846673984 ms)
2025-02-17 15:07:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846724414 ms)
2025-02-17 15:07:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846724415 ms)
2025-02-17 15:07:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:07:25.402636+00:00', 'runtime': '846720.003', 'adc_vb_f': '14.194', 'adc_va_f': '20.755', 'adc_vl_f': '14.188', 'adc_ic_f': '6.549', 'adc_il_f': '0.336', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.178', 'vb_ref': '14.207', 'ahc_r': '2637.6', 'ahc_t': '2637.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '92.917', 'sweep_vmp': '20.792', 'sweep_pmax': '92.041', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '14.194', 'ahc_daily': '7.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846724416 ms)
2025-02-17 15:07:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:06:25.402639+00:00', 'runtime': '846660.003', 'adc_vb_f': '14.117', 'adc_va_f': '20.755', 'adc_vl_f': '14.111', 'adc_ic_f': '6.527', 'adc_il_f': '0.345', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.124', 'vb_ref': '14.207', 'ahc_r': '2637.5', 'ahc_t': '2637.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '92.071', 'sweep_vmp': '20.792', 'sweep_pmax': '92.041', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '14.163', 'ahc_daily': '7.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846724416 ms)
2025-02-17 15:07:35.744 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846733563 ms)
2025-02-17 15:07:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846733564 ms)
2025-02-17 15:07:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:07:25.400044+00:00'} (T+846733565 ms)
2025-02-17 15:08:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846784408 ms)
2025-02-17 15:08:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846784409 ms)
2025-02-17 15:08:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:08:25.400179+00:00', 'runtime': '846780.001', 'adc_vb_f': '14.218', 'adc_va_f': '20.755', 'adc_vl_f': '14.194', 'adc_ic_f': '6.566', 'adc_il_f': '0.338', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.209', 'vb_ref': '14.207', 'ahc_r': '2637.7', 'ahc_t': '2637.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '93.324', 'sweep_vmp': '20.792', 'sweep_pmax': '92.041', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '14.23', 'ahc_daily': '7.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846784411 ms)
2025-02-17 15:08:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:07:25.402636+00:00', 'runtime': '846720.003', 'adc_vb_f': '14.194', 'adc_va_f': '20.755', 'adc_vl_f': '14.188', 'adc_ic_f': '6.549', 'adc_il_f': '0.336', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.178', 'vb_ref': '14.207', 'ahc_r': '2637.6', 'ahc_t': '2637.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '92.917', 'sweep_vmp': '20.792', 'sweep_pmax': '92.041', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '14.194', 'ahc_daily': '7.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846784411 ms)
2025-02-17 15:08:35.652 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846793471 ms)
2025-02-17 15:08:35.652 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846793472 ms)
2025-02-17 15:08:35.653 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:08:25.400051+00:00'} (T+846793473 ms)
2025-02-17 15:09:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846844416 ms)
2025-02-17 15:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846844416 ms)
2025-02-17 15:09:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:09:25.400196+00:00', 'runtime': '846840.001', 'adc_vb_f': '14.111', 'adc_va_f': '20.85', 'adc_vl_f': '14.105', 'adc_ic_f': '6.614', 'adc_il_f': '0.29', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.191', 'vb_ref': '14.207', 'ahc_r': '2637.8', 'ahc_t': '2637.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '93.203', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '14.233', 'ahc_daily': '7.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846844420 ms)
2025-02-17 15:09:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:08:25.400179+00:00', 'runtime': '846780.001', 'adc_vb_f': '14.218', 'adc_va_f': '20.755', 'adc_vl_f': '14.194', 'adc_ic_f': '6.566', 'adc_il_f': '0.338', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.209', 'vb_ref': '14.207', 'ahc_r': '2637.7', 'ahc_t': '2637.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '93.324', 'sweep_vmp': '20.792', 'sweep_pmax': '92.041', 'sweep_voc': '23.355', 'vb_min_daily': '12.164', 'vb_max_daily': '14.23', 'ahc_daily': '7.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846844421 ms)
2025-02-17 15:09:27.474 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+846845293 ms)
2025-02-17 15:09:27.474 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+846845293 ms)
2025-02-17 15:09:27.475 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39366)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+846845294 ms)
2025-02-17 15:09:35.891 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846853710 ms)
2025-02-17 15:09:35.892 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846853711 ms)
2025-02-17 15:09:35.892 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:09:25.400144+00:00'} (T+846853712 ms)
2025-02-17 15:10:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846904407 ms)
2025-02-17 15:10:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846904408 ms)
2025-02-17 15:10:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:10:25.400244+00:00', 'runtime': '846900.001', 'adc_vb_f': '14.27', 'adc_va_f': '20.85', 'adc_vl_f': '14.264', 'adc_ic_f': '6.607', 'adc_il_f': '0.309', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.236', 'vb_ref': '14.207', 'ahc_r': '2637.9', 'ahc_t': '2637.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '94.306', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '14.27', 'ahc_daily': '7.9', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846904412 ms)
2025-02-17 15:10:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:09:25.400196+00:00', 'runtime': '846840.001', 'adc_vb_f': '14.111', 'adc_va_f': '20.85', 'adc_vl_f': '14.105', 'adc_ic_f': '6.614', 'adc_il_f': '0.29', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.191', 'vb_ref': '14.207', 'ahc_r': '2637.8', 'ahc_t': '2637.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '93.203', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '14.233', 'ahc_daily': '7.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846904413 ms)
2025-02-17 15:10:35.721 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846913540 ms)
2025-02-17 15:10:35.722 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846913541 ms)
2025-02-17 15:10:35.722 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:10:25.400441+00:00'} (T+846913542 ms)
2025-02-17 15:11:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+846964410 ms)
2025-02-17 15:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+846964411 ms)
2025-02-17 15:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:11:25.400199+00:00', 'runtime': '846960.001', 'adc_vb_f': '14.294', 'adc_va_f': '20.85', 'adc_vl_f': '14.282', 'adc_ic_f': '6.646', 'adc_il_f': '0.304', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.27', 'vb_ref': '14.178', 'ahc_r': '2638.1', 'ahc_t': '2638.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '95.0', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '14.297', 'ahc_daily': '8.1', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846964412 ms)
2025-02-17 15:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:10:25.400244+00:00', 'runtime': '846900.001', 'adc_vb_f': '14.27', 'adc_va_f': '20.85', 'adc_vl_f': '14.264', 'adc_ic_f': '6.607', 'adc_il_f': '0.309', 't_hs': '12', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.236', 'vb_ref': '14.207', 'ahc_r': '2637.9', 'ahc_t': '2637.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '94.306', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '14.27', 'ahc_daily': '7.9', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+846964412 ms)
2025-02-17 15:11:35.665 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846973484 ms)
2025-02-17 15:11:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+846973485 ms)
2025-02-17 15:11:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:11:25.400114+00:00'} (T+846973486 ms)
2025-02-17 15:12:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847024418 ms)
2025-02-17 15:12:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847024418 ms)
2025-02-17 15:12:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:12:25.400158+00:00', 'runtime': '847020.001', 'adc_vb_f': '14.008', 'adc_va_f': '20.85', 'adc_vl_f': '13.95', 'adc_ic_f': '6.624', 'adc_il_f': '0.285', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.291', 'vb_ref': '14.178', 'ahc_r': '2638.2', 'ahc_t': '2638.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '94.804', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.337', 'ahc_daily': '8.2', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847024419 ms)
2025-02-17 15:12:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:11:25.400199+00:00', 'runtime': '846960.001', 'adc_vb_f': '14.294', 'adc_va_f': '20.85', 'adc_vl_f': '14.282', 'adc_ic_f': '6.646', 'adc_il_f': '0.304', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.27', 'vb_ref': '14.178', 'ahc_r': '2638.1', 'ahc_t': '2638.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '95.0', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.331', 'vb_min_daily': '12.164', 'vb_max_daily': '14.297', 'ahc_daily': '8.1', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847024420 ms)
2025-02-17 15:13:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847084411 ms)
2025-02-17 15:13:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847084412 ms)
2025-02-17 15:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:13:25.400166+00:00', 'runtime': '847080.001', 'adc_vb_f': '14.386', 'adc_va_f': '20.703', 'adc_vl_f': '14.377', 'adc_ic_f': '6.682', 'adc_il_f': '0.292', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.355', 'vb_ref': '14.178', 'ahc_r': '2638.3', 'ahc_t': '2638.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.118', 'sweep_vmp': '20.706', 'sweep_pmax': '93.943', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.386', 'ahc_daily': '8.3', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847084413 ms)
2025-02-17 15:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:12:25.400158+00:00', 'runtime': '847020.001', 'adc_vb_f': '14.008', 'adc_va_f': '20.85', 'adc_vl_f': '13.95', 'adc_ic_f': '6.624', 'adc_il_f': '0.285', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.291', 'vb_ref': '14.178', 'ahc_r': '2638.2', 'ahc_t': '2638.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '94.804', 'sweep_vmp': '20.88', 'sweep_pmax': '92.705', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.337', 'ahc_daily': '8.2', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847084413 ms)
2025-02-17 15:14:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847144406 ms)
2025-02-17 15:14:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847144407 ms)
2025-02-17 15:14:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:14:25.400167+00:00', 'runtime': '847140.001', 'adc_vb_f': '14.423', 'adc_va_f': '20.703', 'adc_vl_f': '14.417', 'adc_ic_f': '6.704', 'adc_il_f': '0.302', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.404', 'vb_ref': '14.178', 'ahc_r': '2638.4', 'ahc_t': '2638.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.752', 'sweep_vmp': '20.706', 'sweep_pmax': '93.943', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.423', 'ahc_daily': '8.4', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847144408 ms)
2025-02-17 15:14:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:13:25.400166+00:00', 'runtime': '847080.001', 'adc_vb_f': '14.386', 'adc_va_f': '20.703', 'adc_vl_f': '14.377', 'adc_ic_f': '6.682', 'adc_il_f': '0.292', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.355', 'vb_ref': '14.178', 'ahc_r': '2638.3', 'ahc_t': '2638.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.118', 'sweep_vmp': '20.706', 'sweep_pmax': '93.943', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.386', 'ahc_daily': '8.3', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847144408 ms)
2025-02-17 15:14:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847153439 ms)
2025-02-17 15:14:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847153439 ms)
2025-02-17 15:14:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:14:25.400343+00:00'} (T+847153440 ms)
2025-02-17 15:15:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847204415 ms)
2025-02-17 15:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847204416 ms)
2025-02-17 15:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:15:25.402791+00:00', 'runtime': '847200.003', 'adc_vb_f': '14.459', 'adc_va_f': '20.703', 'adc_vl_f': '14.426', 'adc_ic_f': '6.704', 'adc_il_f': '0.302', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.432', 'vb_ref': '14.178', 'ahc_r': '2638.5', 'ahc_t': '2638.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.903', 'sweep_vmp': '20.706', 'sweep_pmax': '93.943', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.462', 'ahc_daily': '8.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847204417 ms)
2025-02-17 15:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:14:25.400167+00:00', 'runtime': '847140.001', 'adc_vb_f': '14.423', 'adc_va_f': '20.703', 'adc_vl_f': '14.417', 'adc_ic_f': '6.704', 'adc_il_f': '0.302', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.404', 'vb_ref': '14.178', 'ahc_r': '2638.4', 'ahc_t': '2638.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.752', 'sweep_vmp': '20.706', 'sweep_pmax': '93.943', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.423', 'ahc_daily': '8.4', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847204417 ms)
2025-02-17 15:15:35.621 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847213440 ms)
2025-02-17 15:15:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847213441 ms)
2025-02-17 15:15:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:15:25.400191+00:00'} (T+847213442 ms)
2025-02-17 15:16:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847264410 ms)
2025-02-17 15:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847264410 ms)
2025-02-17 15:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:16:25.402793+00:00', 'runtime': '847260.003', 'adc_vb_f': '14.441', 'adc_va_f': '20.551', 'adc_vl_f': '14.417', 'adc_ic_f': '6.682', 'adc_il_f': '0.319', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '14.178', 'ahc_r': '2638.6', 'ahc_t': '2638.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.48', 'sweep_vmp': '20.563', 'sweep_pmax': '96.163', 'sweep_voc': '23.248', 'vb_min_daily': '12.164', 'vb_max_daily': '14.462', 'ahc_daily': '8.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847264411 ms)
2025-02-17 15:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:15:25.402791+00:00', 'runtime': '847200.003', 'adc_vb_f': '14.459', 'adc_va_f': '20.703', 'adc_vl_f': '14.426', 'adc_ic_f': '6.704', 'adc_il_f': '0.302', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.432', 'vb_ref': '14.178', 'ahc_r': '2638.5', 'ahc_t': '2638.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.903', 'sweep_vmp': '20.706', 'sweep_pmax': '93.943', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.462', 'ahc_daily': '8.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847264412 ms)
2025-02-17 15:16:35.622 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847273441 ms)
2025-02-17 15:16:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847273442 ms)
2025-02-17 15:16:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:16:25.400197+00:00'} (T+847273446 ms)
2025-02-17 15:17:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847324420 ms)
2025-02-17 15:17:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847324420 ms)
2025-02-17 15:17:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:17:25.402801+00:00', 'runtime': '847320.003', 'adc_vb_f': '14.496', 'adc_va_f': '20.554', 'adc_vl_f': '14.453', 'adc_ic_f': '6.682', 'adc_il_f': '0.297', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.465', 'vb_ref': '14.178', 'ahc_r': '2638.7', 'ahc_t': '2638.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.721', 'sweep_vmp': '20.563', 'sweep_pmax': '96.163', 'sweep_voc': '23.248', 'vb_min_daily': '12.164', 'vb_max_daily': '14.499', 'ahc_daily': '8.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847324421 ms)
2025-02-17 15:17:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:16:25.402793+00:00', 'runtime': '847260.003', 'adc_vb_f': '14.441', 'adc_va_f': '20.551', 'adc_vl_f': '14.417', 'adc_ic_f': '6.682', 'adc_il_f': '0.319', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '14.178', 'ahc_r': '2638.6', 'ahc_t': '2638.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.48', 'sweep_vmp': '20.563', 'sweep_pmax': '96.163', 'sweep_voc': '23.248', 'vb_min_daily': '12.164', 'vb_max_daily': '14.462', 'ahc_daily': '8.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847324422 ms)
2025-02-17 15:17:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847333435 ms)
2025-02-17 15:17:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847333436 ms)
2025-02-17 15:17:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:17:25.400164+00:00'} (T+847333438 ms)
2025-02-17 15:17:40.186 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+847338005 ms)
2025-02-17 15:17:40.186 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+847338006 ms)
2025-02-17 15:17:40.187 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39902)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847338006 ms)
2025-02-17 15:18:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847384412 ms)
2025-02-17 15:18:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847384413 ms)
2025-02-17 15:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:18:25.400537+00:00', 'runtime': '847380.001', 'adc_vb_f': '14.529', 'adc_va_f': '20.554', 'adc_vl_f': '14.493', 'adc_ic_f': '6.682', 'adc_il_f': '0.309', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.514', 'vb_ref': '14.178', 'ahc_r': '2638.8', 'ahc_t': '2638.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '97.069', 'sweep_vmp': '20.563', 'sweep_pmax': '96.163', 'sweep_voc': '23.248', 'vb_min_daily': '12.164', 'vb_max_daily': '14.539', 'ahc_daily': '8.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847384417 ms)
2025-02-17 15:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:17:25.402801+00:00', 'runtime': '847320.003', 'adc_vb_f': '14.496', 'adc_va_f': '20.554', 'adc_vl_f': '14.453', 'adc_ic_f': '6.682', 'adc_il_f': '0.297', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.465', 'vb_ref': '14.178', 'ahc_r': '2638.7', 'ahc_t': '2638.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '96.721', 'sweep_vmp': '20.563', 'sweep_pmax': '96.163', 'sweep_voc': '23.248', 'vb_min_daily': '12.164', 'vb_max_daily': '14.499', 'ahc_daily': '8.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847384418 ms)
2025-02-17 15:18:35.738 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847393557 ms)
2025-02-17 15:18:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847393558 ms)
2025-02-17 15:18:35.740 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:18:25.401132+00:00'} (T+847393559 ms)
2025-02-17 15:19:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847444408 ms)
2025-02-17 15:19:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847444408 ms)
2025-02-17 15:19:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:19:25.402675+00:00', 'runtime': '847440.003', 'adc_vb_f': '14.59', 'adc_va_f': '20.554', 'adc_vl_f': '14.569', 'adc_ic_f': '6.735', 'adc_il_f': '0.292', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.551', 'vb_ref': '14.178', 'ahc_r': '2638.9', 'ahc_t': '2638.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '98.277', 'sweep_vmp': '20.572', 'sweep_pmax': '96.933', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.606', 'ahc_daily': '8.9', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847444410 ms)
2025-02-17 15:19:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:18:25.400537+00:00', 'runtime': '847380.001', 'adc_vb_f': '14.529', 'adc_va_f': '20.554', 'adc_vl_f': '14.493', 'adc_ic_f': '6.682', 'adc_il_f': '0.309', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.514', 'vb_ref': '14.178', 'ahc_r': '2638.8', 'ahc_t': '2638.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '97.069', 'sweep_vmp': '20.563', 'sweep_pmax': '96.163', 'sweep_voc': '23.248', 'vb_min_daily': '12.164', 'vb_max_daily': '14.539', 'ahc_daily': '8.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847444410 ms)
2025-02-17 15:19:35.638 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847453457 ms)
2025-02-17 15:19:35.639 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847453458 ms)
2025-02-17 15:19:35.640 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:19:25.400070+00:00'} (T+847453459 ms)
2025-02-17 15:20:04.437 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+847482256 ms)
2025-02-17 15:20:04.437 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+847482256 ms)
2025-02-17 15:20:04.438 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40058)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847482257 ms)
2025-02-17 15:20:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847504414 ms)
2025-02-17 15:20:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847504415 ms)
2025-02-17 15:20:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:20:25.402754+00:00', 'runtime': '847500.003', 'adc_vb_f': '14.651', 'adc_va_f': '20.67', 'adc_vl_f': '14.606', 'adc_ic_f': '6.718', 'adc_il_f': '0.256', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.618', 'vb_ref': '14.178', 'ahc_r': '2639.1', 'ahc_t': '2639.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '97.129', 'sweep_vmp': '20.572', 'sweep_pmax': '96.933', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.713', 'ahc_daily': '9.1', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847504417 ms)
2025-02-17 15:20:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:19:25.402675+00:00', 'runtime': '847440.003', 'adc_vb_f': '14.59', 'adc_va_f': '20.554', 'adc_vl_f': '14.569', 'adc_ic_f': '6.735', 'adc_il_f': '0.292', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.551', 'vb_ref': '14.178', 'ahc_r': '2638.9', 'ahc_t': '2638.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '98.277', 'sweep_vmp': '20.572', 'sweep_pmax': '96.933', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.606', 'ahc_daily': '8.9', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847504417 ms)
2025-02-17 15:20:35.738 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847513558 ms)
2025-02-17 15:20:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847513558 ms)
2025-02-17 15:20:35.740 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:20:25.400155+00:00'} (T+847513559 ms)
2025-02-17 15:21:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847564409 ms)
2025-02-17 15:21:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847564410 ms)
2025-02-17 15:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:21:25.400540+00:00', 'runtime': '847560.001', 'adc_vb_f': '14.658', 'adc_va_f': '20.911', 'adc_vl_f': '14.645', 'adc_ic_f': '6.713', 'adc_il_f': '0.261', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.648', 'vb_ref': '14.178', 'ahc_r': '2639.2', 'ahc_t': '2639.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '98.246', 'sweep_vmp': '20.572', 'sweep_pmax': '96.933', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.2', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847564414 ms)
2025-02-17 15:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:20:25.402754+00:00', 'runtime': '847500.003', 'adc_vb_f': '14.651', 'adc_va_f': '20.67', 'adc_vl_f': '14.606', 'adc_ic_f': '6.718', 'adc_il_f': '0.256', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.618', 'vb_ref': '14.178', 'ahc_r': '2639.1', 'ahc_t': '2639.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '97.129', 'sweep_vmp': '20.572', 'sweep_pmax': '96.933', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.713', 'ahc_daily': '9.1', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847564414 ms)
2025-02-17 15:21:32.542 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+847570361 ms)
2025-02-17 15:21:32.542 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+847570361 ms)
2025-02-17 15:21:32.542 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847570361 ms)
2025-02-17 15:21:32.543 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+847570362 ms)
2025-02-17 15:21:32.543 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+847570362 ms)
2025-02-17 15:21:32.543 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847570362 ms)
2025-02-17 15:21:32.544 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+847570363 ms)
2025-02-17 15:21:32.544 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+847570363 ms)
2025-02-17 15:21:32.544 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847570363 ms)
2025-02-17 15:21:32.545 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+847570364 ms)
2025-02-17 15:21:32.545 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+847570364 ms)
2025-02-17 15:21:32.551 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847570370 ms)
2025-02-17 15:21:32.552 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+847570371 ms)
2025-02-17 15:21:32.555 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+847570374 ms)
2025-02-17 15:21:32.557 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847570377 ms)
2025-02-17 15:21:35.871 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847573690 ms)
2025-02-17 15:21:35.871 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847573691 ms)
2025-02-17 15:21:35.876 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:21:25.400079+00:00'} (T+847573695 ms)
2025-02-17 15:21:48.632 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+847586451 ms)
2025-02-17 15:21:48.632 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+847586452 ms)
2025-02-17 15:21:48.633 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40172)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847586452 ms)
2025-02-17 15:22:20.744 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+847618563 ms)
2025-02-17 15:22:20.744 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+847618564 ms)
2025-02-17 15:22:20.745 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40206)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847618564 ms)
2025-02-17 15:22:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847624403 ms)
2025-02-17 15:22:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847624404 ms)
2025-02-17 15:22:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:22:25.400217+00:00', 'runtime': '847620.001', 'adc_vb_f': '14.651', 'adc_va_f': '20.914', 'adc_vl_f': '14.645', 'adc_ic_f': '6.697', 'adc_il_f': '0.283', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.642', 'vb_ref': '14.178', 'ahc_r': '2639.3', 'ahc_t': '2639.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '98.065', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.3', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847624407 ms)
2025-02-17 15:22:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:21:25.400540+00:00', 'runtime': '847560.001', 'adc_vb_f': '14.658', 'adc_va_f': '20.911', 'adc_vl_f': '14.645', 'adc_ic_f': '6.713', 'adc_il_f': '0.261', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.648', 'vb_ref': '14.178', 'ahc_r': '2639.2', 'ahc_t': '2639.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '98.246', 'sweep_vmp': '20.572', 'sweep_pmax': '96.933', 'sweep_voc': '23.267', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.2', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847624408 ms)
2025-02-17 15:22:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847633420 ms)
2025-02-17 15:22:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847633421 ms)
2025-02-17 15:22:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:22:25.400456+00:00'} (T+847633423 ms)
2025-02-17 15:23:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847684412 ms)
2025-02-17 15:23:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847684412 ms)
2025-02-17 15:23:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:23:25.400183+00:00', 'runtime': '847680.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.021', 'adc_vl_f': '14.645', 'adc_ic_f': '6.687', 'adc_il_f': '0.273', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2639.4', 'ahc_t': '2639.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '97.929', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.4', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847684413 ms)
2025-02-17 15:23:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:22:25.400217+00:00', 'runtime': '847620.001', 'adc_vb_f': '14.651', 'adc_va_f': '20.914', 'adc_vl_f': '14.645', 'adc_ic_f': '6.697', 'adc_il_f': '0.283', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.642', 'vb_ref': '14.178', 'ahc_r': '2639.3', 'ahc_t': '2639.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '98.065', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.3', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847684416 ms)
2025-02-17 15:24:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847744409 ms)
2025-02-17 15:24:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847744409 ms)
2025-02-17 15:24:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:24:25.400192+00:00', 'runtime': '847740.001', 'adc_vb_f': '14.664', 'adc_va_f': '21.109', 'adc_vl_f': '14.645', 'adc_ic_f': '6.636', 'adc_il_f': '0.285', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.658', 'vb_ref': '14.178', 'ahc_r': '2639.5', 'ahc_t': '2639.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.755', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847744410 ms)
2025-02-17 15:24:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:23:25.400183+00:00', 'runtime': '847680.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.021', 'adc_vl_f': '14.645', 'adc_ic_f': '6.687', 'adc_il_f': '0.273', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2639.4', 'ahc_t': '2639.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '97.929', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.4', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847744410 ms)
2025-02-17 15:25:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847804417 ms)
2025-02-17 15:25:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847804417 ms)
2025-02-17 15:25:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:25:25.400192+00:00', 'runtime': '847800.001', 'adc_vb_f': '14.658', 'adc_va_f': '21.011', 'adc_vl_f': '14.636', 'adc_ic_f': '6.783', 'adc_il_f': '0.307', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2639.6', 'ahc_t': '2639.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '100.526', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847804418 ms)
2025-02-17 15:25:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:24:25.400192+00:00', 'runtime': '847740.001', 'adc_vb_f': '14.664', 'adc_va_f': '21.109', 'adc_vl_f': '14.645', 'adc_ic_f': '6.636', 'adc_il_f': '0.285', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.658', 'vb_ref': '14.178', 'ahc_r': '2639.5', 'ahc_t': '2639.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.755', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.5', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847804419 ms)
2025-02-17 15:25:36.234 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847814054 ms)
2025-02-17 15:25:36.235 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847814054 ms)
2025-02-17 15:25:36.242 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:25:25.400344+00:00'} (T+847814061 ms)
2025-02-17 15:26:17.101 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+847854921 ms)
2025-02-17 15:26:17.102 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+847854921 ms)
2025-02-17 15:26:17.102 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40468)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+847854921 ms)
2025-02-17 15:26:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847864411 ms)
2025-02-17 15:26:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847864412 ms)
2025-02-17 15:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:26:25.400186+00:00', 'runtime': '847860.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.225', 'adc_vl_f': '14.624', 'adc_ic_f': '6.556', 'adc_il_f': '0.326', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.658', 'vb_ref': '14.178', 'ahc_r': '2639.7', 'ahc_t': '2639.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '98.986', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847864413 ms)
2025-02-17 15:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:25:25.400192+00:00', 'runtime': '847800.001', 'adc_vb_f': '14.658', 'adc_va_f': '21.011', 'adc_vl_f': '14.636', 'adc_ic_f': '6.783', 'adc_il_f': '0.307', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2639.6', 'ahc_t': '2639.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '100.526', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.6', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847864414 ms)
2025-02-17 15:26:35.624 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847873443 ms)
2025-02-17 15:26:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+847873444 ms)
2025-02-17 15:26:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:26:25.400047+00:00'} (T+847873447 ms)
2025-02-17 15:27:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847924422 ms)
2025-02-17 15:27:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847924422 ms)
2025-02-17 15:27:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:27:25.401126+00:00', 'runtime': '847920.002', 'adc_vb_f': '14.651', 'adc_va_f': '21.127', 'adc_vl_f': '14.609', 'adc_ic_f': '6.607', 'adc_il_f': '0.309', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2639.8', 'ahc_t': '2639.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.797', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847924423 ms)
2025-02-17 15:27:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:26:25.400186+00:00', 'runtime': '847860.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.225', 'adc_vl_f': '14.624', 'adc_ic_f': '6.556', 'adc_il_f': '0.326', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.658', 'vb_ref': '14.178', 'ahc_r': '2639.7', 'ahc_t': '2639.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '98.986', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.7', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847924424 ms)
2025-02-17 15:28:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+847984416 ms)
2025-02-17 15:28:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+847984417 ms)
2025-02-17 15:28:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:28:25.400162+00:00', 'runtime': '847980.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.127', 'adc_vl_f': '14.645', 'adc_ic_f': '6.597', 'adc_il_f': '0.287', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2640.0', 'ahc_t': '2640.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.616', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.0', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847984418 ms)
2025-02-17 15:28:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:27:25.401126+00:00', 'runtime': '847920.002', 'adc_vb_f': '14.651', 'adc_va_f': '21.127', 'adc_vl_f': '14.609', 'adc_ic_f': '6.607', 'adc_il_f': '0.309', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2639.8', 'ahc_t': '2639.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.797', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '9.8', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+847984418 ms)
2025-02-17 15:29:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848044407 ms)
2025-02-17 15:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848044408 ms)
2025-02-17 15:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:29:25.400164+00:00', 'runtime': '848040.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.155', 'adc_vl_f': '14.606', 'adc_ic_f': '6.583', 'adc_il_f': '0.304', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.639', 'vb_ref': '14.148', 'ahc_r': '2640.1', 'ahc_t': '2640.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.812', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848044409 ms)
2025-02-17 15:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:28:25.400162+00:00', 'runtime': '847980.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.127', 'adc_vl_f': '14.645', 'adc_ic_f': '6.597', 'adc_il_f': '0.287', 't_hs': '13', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2640.0', 'ahc_t': '2640.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1590.9', 'ahl_t': '1590.9', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.616', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.0', 'ahl_daily': '6.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848044409 ms)
2025-02-17 15:29:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848053427 ms)
2025-02-17 15:29:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848053427 ms)
2025-02-17 15:29:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:29:25.400331+00:00'} (T+848053428 ms)
2025-02-17 15:30:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848104403 ms)
2025-02-17 15:30:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848104403 ms)
2025-02-17 15:30:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:30:25.400216+00:00', 'runtime': '848100.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.234', 'adc_vl_f': '14.606', 'adc_ic_f': '6.556', 'adc_il_f': '0.292', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.2', 'ahc_t': '2640.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.314', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848104404 ms)
2025-02-17 15:30:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:29:25.400164+00:00', 'runtime': '848040.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.155', 'adc_vl_f': '14.606', 'adc_ic_f': '6.583', 'adc_il_f': '0.304', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.639', 'vb_ref': '14.148', 'ahc_r': '2640.1', 'ahc_t': '2640.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.812', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848104405 ms)
2025-02-17 15:30:35.778 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848113598 ms)
2025-02-17 15:30:35.779 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848113598 ms)
2025-02-17 15:30:35.780 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:30:25.400225+00:00'} (T+848113599 ms)
2025-02-17 15:31:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848164413 ms)
2025-02-17 15:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848164413 ms)
2025-02-17 15:31:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:31:25.400214+00:00', 'runtime': '848160.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.237', 'adc_vl_f': '14.606', 'adc_ic_f': '6.549', 'adc_il_f': '0.29', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.3', 'ahc_t': '2640.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.178', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848164415 ms)
2025-02-17 15:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:30:25.400216+00:00', 'runtime': '848100.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.234', 'adc_vl_f': '14.606', 'adc_ic_f': '6.556', 'adc_il_f': '0.292', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.2', 'ahc_t': '2640.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.314', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848164415 ms)
2025-02-17 15:31:36.146 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848173965 ms)
2025-02-17 15:31:36.146 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848173966 ms)
2025-02-17 15:31:36.147 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:31:25.400012+00:00'} (T+848173966 ms)
2025-02-17 15:32:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848224405 ms)
2025-02-17 15:32:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848224406 ms)
2025-02-17 15:32:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:32:25.400267+00:00', 'runtime': '848220.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.513', 'adc_il_f': '0.271', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.4', 'ahc_t': '2640.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.072', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848224407 ms)
2025-02-17 15:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:31:25.400214+00:00', 'runtime': '848160.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.237', 'adc_vl_f': '14.606', 'adc_ic_f': '6.549', 'adc_il_f': '0.29', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.3', 'ahc_t': '2640.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.178', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848224407 ms)
2025-02-17 15:32:36.056 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848233876 ms)
2025-02-17 15:32:36.057 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848233876 ms)
2025-02-17 15:32:36.058 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:32:25.400309+00:00'} (T+848233877 ms)
2025-02-17 15:33:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848284416 ms)
2025-02-17 15:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848284417 ms)
2025-02-17 15:33:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:33:25.400165+00:00', 'runtime': '848280.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.489', 'adc_il_f': '0.254', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.5', 'ahc_t': '2640.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.453', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848284418 ms)
2025-02-17 15:33:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:32:25.400267+00:00', 'runtime': '848220.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.513', 'adc_il_f': '0.271', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.4', 'ahc_t': '2640.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '96.072', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848284418 ms)
2025-02-17 15:33:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848293431 ms)
2025-02-17 15:33:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848293431 ms)
2025-02-17 15:33:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:33:25.400235+00:00'} (T+848293432 ms)
2025-02-17 15:34:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848344409 ms)
2025-02-17 15:34:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848344410 ms)
2025-02-17 15:34:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:34:25.400167+00:00', 'runtime': '848340.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.277', 'adc_vl_f': '14.606', 'adc_ic_f': '6.479', 'adc_il_f': '0.261', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2640.6', 'ahc_t': '2640.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.363', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848344411 ms)
2025-02-17 15:34:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:33:25.400165+00:00', 'runtime': '848280.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.489', 'adc_il_f': '0.254', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.5', 'ahc_t': '2640.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.453', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848344412 ms)
2025-02-17 15:34:36.363 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848354182 ms)
2025-02-17 15:34:36.364 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848354183 ms)
2025-02-17 15:34:36.365 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:34:25.400176+00:00'} (T+848354184 ms)
2025-02-17 15:35:07.819 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+848385639 ms)
2025-02-17 15:35:07.820 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+848385639 ms)
2025-02-17 15:35:07.820 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41034)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+848385640 ms)
2025-02-17 15:35:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848404405 ms)
2025-02-17 15:35:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848404405 ms)
2025-02-17 15:35:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:35:25.400179+00:00', 'runtime': '848400.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.259', 'adc_vl_f': '14.606', 'adc_ic_f': '6.467', 'adc_il_f': '0.271', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.7', 'ahc_t': '2640.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.925', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848404406 ms)
2025-02-17 15:35:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:34:25.400167+00:00', 'runtime': '848340.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.277', 'adc_vl_f': '14.606', 'adc_ic_f': '6.479', 'adc_il_f': '0.261', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2640.6', 'ahc_t': '2640.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.363', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848404407 ms)
2025-02-17 15:35:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848413434 ms)
2025-02-17 15:35:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848413435 ms)
2025-02-17 15:35:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:35:25.400465+00:00'} (T+848413436 ms)
2025-02-17 15:36:07.969 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+848445788 ms)
2025-02-17 15:36:07.969 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+848445789 ms)
2025-02-17 15:36:07.970 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41110)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+848445789 ms)
2025-02-17 15:36:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848464415 ms)
2025-02-17 15:36:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848464415 ms)
2025-02-17 15:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:36:25.400187+00:00', 'runtime': '848460.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.329', 'adc_vl_f': '14.606', 'adc_ic_f': '6.491', 'adc_il_f': '0.302', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2640.8', 'ahc_t': '2640.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.242', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848464420 ms)
2025-02-17 15:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:35:25.400179+00:00', 'runtime': '848400.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.259', 'adc_vl_f': '14.606', 'adc_ic_f': '6.467', 'adc_il_f': '0.271', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2640.7', 'ahc_t': '2640.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.925', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848464420 ms)
2025-02-17 15:36:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848473432 ms)
2025-02-17 15:36:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848473433 ms)
2025-02-17 15:36:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:36:25.400542+00:00'} (T+848473434 ms)
2025-02-17 15:37:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848524409 ms)
2025-02-17 15:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848524409 ms)
2025-02-17 15:37:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:37:25.400204+00:00', 'runtime': '848520.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.326', 'adc_vl_f': '14.606', 'adc_ic_f': '6.469', 'adc_il_f': '0.309', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2640.9', 'ahc_t': '2640.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.834', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848524410 ms)
2025-02-17 15:37:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:36:25.400187+00:00', 'runtime': '848460.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.329', 'adc_vl_f': '14.606', 'adc_ic_f': '6.491', 'adc_il_f': '0.302', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2640.8', 'ahc_t': '2640.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '95.242', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848524411 ms)
2025-02-17 15:37:35.701 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848533521 ms)
2025-02-17 15:37:35.702 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848533521 ms)
2025-02-17 15:37:35.703 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:37:25.400241+00:00'} (T+848533522 ms)
2025-02-17 15:38:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848584416 ms)
2025-02-17 15:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848584417 ms)
2025-02-17 15:38:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:38:25.400180+00:00', 'runtime': '848580.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.246', 'adc_vl_f': '14.606', 'adc_ic_f': '6.45', 'adc_il_f': '0.302', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.0', 'ahc_t': '2641.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.487', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848584417 ms)
2025-02-17 15:38:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:37:25.400204+00:00', 'runtime': '848520.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.326', 'adc_vl_f': '14.606', 'adc_ic_f': '6.469', 'adc_il_f': '0.309', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2640.9', 'ahc_t': '2640.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.834', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '10.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848584418 ms)
2025-02-17 15:38:35.658 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848593477 ms)
2025-02-17 15:38:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848593478 ms)
2025-02-17 15:38:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:38:25.400102+00:00'} (T+848593479 ms)
2025-02-17 15:39:03.248 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+848621068 ms)
2025-02-17 15:39:03.249 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+848621068 ms)
2025-02-17 15:39:03.249 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41298)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+848621068 ms)
2025-02-17 15:39:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848644411 ms)
2025-02-17 15:39:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848644411 ms)
2025-02-17 15:39:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:39:25.400181+00:00', 'runtime': '848640.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.32', 'adc_vl_f': '14.606', 'adc_ic_f': '6.416', 'adc_il_f': '0.297', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.1', 'ahc_t': '2641.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.351', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848644413 ms)
2025-02-17 15:39:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:38:25.400180+00:00', 'runtime': '848580.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.246', 'adc_vl_f': '14.606', 'adc_ic_f': '6.45', 'adc_il_f': '0.302', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.0', 'ahc_t': '2641.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.487', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848644414 ms)
2025-02-17 15:39:35.782 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848653602 ms)
2025-02-17 15:39:35.783 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848653602 ms)
2025-02-17 15:39:35.784 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:39:25.400146+00:00'} (T+848653603 ms)
2025-02-17 15:40:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848704420 ms)
2025-02-17 15:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848704420 ms)
2025-02-17 15:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:40:25.400033+00:00', 'runtime': '848700.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.338', 'adc_vl_f': '14.606', 'adc_ic_f': '6.387', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.3', 'ahc_t': '2641.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.094', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848704421 ms)
2025-02-17 15:40:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:39:25.400181+00:00', 'runtime': '848640.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.32', 'adc_vl_f': '14.606', 'adc_ic_f': '6.416', 'adc_il_f': '0.297', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.1', 'ahc_t': '2641.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.351', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848704422 ms)
2025-02-17 15:40:35.769 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848713588 ms)
2025-02-17 15:40:35.769 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848713589 ms)
2025-02-17 15:40:35.771 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:40:25.409128+00:00'} (T+848713590 ms)
2025-02-17 15:41:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848764416 ms)
2025-02-17 15:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848764417 ms)
2025-02-17 15:41:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:41:25.401216+00:00', 'runtime': '848760.002', 'adc_vb_f': '14.627', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.334', 'adc_il_f': '0.249', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.4', 'ahc_t': '2641.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '93.611', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848764419 ms)
2025-02-17 15:41:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:40:25.400033+00:00', 'runtime': '848700.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.338', 'adc_vl_f': '14.606', 'adc_ic_f': '6.387', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.3', 'ahc_t': '2641.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '94.094', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848764419 ms)
2025-02-17 15:42:10.560 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+848808379 ms)
2025-02-17 15:42:10.560 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+848808379 ms)
2025-02-17 15:42:10.560 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41498)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+848808380 ms)
2025-02-17 15:42:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848824410 ms)
2025-02-17 15:42:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848824410 ms)
2025-02-17 15:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:42:25.400181+00:00', 'runtime': '848820.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.35', 'adc_vl_f': '14.606', 'adc_ic_f': '6.339', 'adc_il_f': '0.29', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.5', 'ahc_t': '2641.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '92.766', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848824411 ms)
2025-02-17 15:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:41:25.401216+00:00', 'runtime': '848760.002', 'adc_vb_f': '14.627', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.334', 'adc_il_f': '0.249', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2641.4', 'ahc_t': '2641.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '93.611', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848824411 ms)
2025-02-17 15:43:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848884419 ms)
2025-02-17 15:43:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848884420 ms)
2025-02-17 15:43:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:43:25.402770+00:00', 'runtime': '848880.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.451', 'adc_vl_f': '14.606', 'adc_ic_f': '6.293', 'adc_il_f': '0.271', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.6', 'ahc_t': '2641.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '92.992', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848884421 ms)
2025-02-17 15:43:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:42:25.400181+00:00', 'runtime': '848820.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.35', 'adc_vl_f': '14.606', 'adc_ic_f': '6.339', 'adc_il_f': '0.29', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.5', 'ahc_t': '2641.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '92.766', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848884421 ms)
2025-02-17 15:43:35.870 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848893689 ms)
2025-02-17 15:43:35.870 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+848893689 ms)
2025-02-17 15:43:35.871 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:43:25.400113+00:00'} (T+848893690 ms)
2025-02-17 15:43:39.741 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+848897560 ms)
2025-02-17 15:43:39.742 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+848897561 ms)
2025-02-17 15:43:39.742 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41592)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+848897561 ms)
2025-02-17 15:44:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+848944415 ms)
2025-02-17 15:44:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+848944416 ms)
2025-02-17 15:44:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:44:25.402842+00:00', 'runtime': '848940.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.436', 'adc_vl_f': '14.606', 'adc_ic_f': '6.305', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.7', 'ahc_t': '2641.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '92.252', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848944417 ms)
2025-02-17 15:44:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:43:25.402770+00:00', 'runtime': '848880.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.451', 'adc_vl_f': '14.606', 'adc_ic_f': '6.293', 'adc_il_f': '0.271', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.6', 'ahc_t': '2641.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '92.992', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+848944417 ms)
2025-02-17 15:45:25.922 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+849003741 ms)
2025-02-17 15:45:25.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+849003741 ms)
2025-02-17 15:45:25.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41706)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849003742 ms)
2025-02-17 15:45:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849004409 ms)
2025-02-17 15:45:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849004409 ms)
2025-02-17 15:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:45:25.401017+00:00', 'runtime': '849000.002', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.303', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.8', 'ahc_t': '2641.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.754', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849004410 ms)
2025-02-17 15:45:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:44:25.402842+00:00', 'runtime': '848940.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.436', 'adc_vl_f': '14.606', 'adc_ic_f': '6.305', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.7', 'ahc_t': '2641.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '92.252', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849004411 ms)
2025-02-17 15:46:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849064424 ms)
2025-02-17 15:46:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849064424 ms)
2025-02-17 15:46:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:46:25.400258+00:00', 'runtime': '849060.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.276', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.9', 'ahc_t': '2641.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.724', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849064425 ms)
2025-02-17 15:46:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:45:25.401017+00:00', 'runtime': '849000.002', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.303', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.8', 'ahc_t': '2641.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.754', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849064425 ms)
2025-02-17 15:46:48.024 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085843 ms)
2025-02-17 15:46:48.024 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085843 ms)
2025-02-17 15:46:48.024 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085843 ms)
2025-02-17 15:46:48.025 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085844 ms)
2025-02-17 15:46:48.025 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085844 ms)
2025-02-17 15:46:48.025 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085844 ms)
2025-02-17 15:46:48.026 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085845 ms)
2025-02-17 15:46:48.026 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085845 ms)
2025-02-17 15:46:48.026 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085845 ms)
2025-02-17 15:46:48.026 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085846 ms)
2025-02-17 15:46:48.026 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085846 ms)
2025-02-17 15:46:48.027 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085846 ms)
2025-02-17 15:46:48.027 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085846 ms)
2025-02-17 15:46:48.027 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085846 ms)
2025-02-17 15:46:48.027 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085847 ms)
2025-02-17 15:46:48.028 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085847 ms)
2025-02-17 15:46:48.028 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085847 ms)
2025-02-17 15:46:48.028 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085847 ms)
2025-02-17 15:46:48.029 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085848 ms)
2025-02-17 15:46:48.029 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085848 ms)
2025-02-17 15:46:48.029 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085848 ms)
2025-02-17 15:46:48.031 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085851 ms)
2025-02-17 15:46:48.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085851 ms)
2025-02-17 15:46:48.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085852 ms)
2025-02-17 15:46:48.033 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085853 ms)
2025-02-17 15:46:48.034 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085853 ms)
2025-02-17 15:46:48.034 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085854 ms)
2025-02-17 15:46:48.035 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+849085855 ms)
2025-02-17 15:46:48.036 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+849085855 ms)
2025-02-17 15:46:48.036 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849085856 ms)
2025-02-17 15:47:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849124414 ms)
2025-02-17 15:47:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849124414 ms)
2025-02-17 15:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:47:25.400659+00:00', 'runtime': '849120.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.347', 'adc_vl_f': '14.606', 'adc_ic_f': '6.271', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2642.0', 'ahc_t': '2642.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.679', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849124415 ms)
2025-02-17 15:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:46:25.400258+00:00', 'runtime': '849060.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.276', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2641.9', 'ahc_t': '2641.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.724', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '11.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849124416 ms)
2025-02-17 15:48:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849184406 ms)
2025-02-17 15:48:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849184406 ms)
2025-02-17 15:48:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:48:25.400178+00:00', 'runtime': '849180.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.353', 'adc_vl_f': '14.606', 'adc_ic_f': '6.238', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.1', 'ahc_t': '2642.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.618', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849184407 ms)
2025-02-17 15:48:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:47:25.400659+00:00', 'runtime': '849120.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.347', 'adc_vl_f': '14.606', 'adc_ic_f': '6.271', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2642.0', 'ahc_t': '2642.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.679', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849184408 ms)
2025-02-17 15:49:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849244415 ms)
2025-02-17 15:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849244416 ms)
2025-02-17 15:49:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:49:25.402794+00:00', 'runtime': '849240.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.359', 'adc_vl_f': '14.606', 'adc_ic_f': '6.242', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.2', 'ahc_t': '2642.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.316', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849244417 ms)
2025-02-17 15:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:48:25.400178+00:00', 'runtime': '849180.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.353', 'adc_vl_f': '14.606', 'adc_ic_f': '6.238', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.1', 'ahc_t': '2642.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.618', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849244417 ms)
2025-02-17 15:50:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849304408 ms)
2025-02-17 15:50:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849304408 ms)
2025-02-17 15:50:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:50:25.400824+00:00', 'runtime': '849300.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.442', 'adc_vl_f': '14.606', 'adc_ic_f': '6.177', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.3', 'ahc_t': '2642.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '90.712', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849304409 ms)
2025-02-17 15:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:49:25.402794+00:00', 'runtime': '849240.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.359', 'adc_vl_f': '14.606', 'adc_ic_f': '6.242', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.2', 'ahc_t': '2642.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '91.316', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849304410 ms)
2025-02-17 15:51:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849364417 ms)
2025-02-17 15:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849364417 ms)
2025-02-17 15:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:51:25.400181+00:00', 'runtime': '849360.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.448', 'adc_vl_f': '14.606', 'adc_ic_f': '6.109', 'adc_il_f': '0.256', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.4', 'ahc_t': '2642.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '90.516', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849364418 ms)
2025-02-17 15:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:50:25.400824+00:00', 'runtime': '849300.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.442', 'adc_vl_f': '14.606', 'adc_ic_f': '6.177', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.3', 'ahc_t': '2642.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '90.712', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849364419 ms)
2025-02-17 15:51:35.931 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+849373751 ms)
2025-02-17 15:51:35.932 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+849373751 ms)
2025-02-17 15:51:35.934 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:51:25.400329+00:00'} (T+849373753 ms)
2025-02-17 15:52:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849424412 ms)
2025-02-17 15:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849424413 ms)
2025-02-17 15:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:52:25.402780+00:00', 'runtime': '849420.003', 'adc_vb_f': '14.627', 'adc_va_f': '21.429', 'adc_vl_f': '14.606', 'adc_ic_f': '6.076', 'adc_il_f': '0.263', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.5', 'ahc_t': '2642.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '90.712', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849424414 ms)
2025-02-17 15:52:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:51:25.400181+00:00', 'runtime': '849360.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.448', 'adc_vl_f': '14.606', 'adc_ic_f': '6.109', 'adc_il_f': '0.256', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.4', 'ahc_t': '2642.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '90.516', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849424414 ms)
2025-02-17 15:53:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849484406 ms)
2025-02-17 15:53:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849484407 ms)
2025-02-17 15:53:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:53:25.400015+00:00', 'runtime': '849480.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.076', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2642.6', 'ahc_t': '2642.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '88.719', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849484408 ms)
2025-02-17 15:53:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:52:25.402780+00:00', 'runtime': '849420.003', 'adc_vb_f': '14.627', 'adc_va_f': '21.429', 'adc_vl_f': '14.606', 'adc_ic_f': '6.076', 'adc_il_f': '0.263', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.5', 'ahc_t': '2642.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '90.712', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849484408 ms)
2025-02-17 15:53:54.651 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+849512471 ms)
2025-02-17 15:53:54.652 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+849512471 ms)
2025-02-17 15:53:54.652 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42252)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+849512471 ms)
2025-02-17 15:54:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849544415 ms)
2025-02-17 15:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849544415 ms)
2025-02-17 15:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:54:25.400576+00:00', 'runtime': '849540.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.35', 'adc_vl_f': '14.606', 'adc_ic_f': '6.018', 'adc_il_f': '0.256', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.7', 'ahc_t': '2642.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '88.689', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849544416 ms)
2025-02-17 15:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:53:25.400015+00:00', 'runtime': '849480.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.344', 'adc_vl_f': '14.606', 'adc_ic_f': '6.076', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2642.6', 'ahc_t': '2642.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20122', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '88.719', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849544417 ms)
2025-02-17 15:55:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849604408 ms)
2025-02-17 15:55:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849604408 ms)
2025-02-17 15:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:55:25.400162+00:00', 'runtime': '849600.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.408', 'adc_vl_f': '14.578', 'adc_ic_f': '6.027', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.624', 'vb_ref': '14.119', 'ahc_r': '2642.8', 'ahc_t': '2642.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '88.81', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849604409 ms)
2025-02-17 15:55:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:54:25.400576+00:00', 'runtime': '849540.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.35', 'adc_vl_f': '14.606', 'adc_ic_f': '6.018', 'adc_il_f': '0.256', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2642.7', 'ahc_t': '2642.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '88.689', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849604410 ms)
2025-02-17 15:56:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849664403 ms)
2025-02-17 15:56:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849664404 ms)
2025-02-17 15:56:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:56:25.400179+00:00', 'runtime': '849660.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.581', 'adc_ic_f': '5.989', 'adc_il_f': '0.285', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.603', 'vb_ref': '14.119', 'ahc_r': '2642.9', 'ahc_t': '2642.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '87.149', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849664405 ms)
2025-02-17 15:56:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:55:25.400162+00:00', 'runtime': '849600.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.408', 'adc_vl_f': '14.578', 'adc_ic_f': '6.027', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.624', 'vb_ref': '14.119', 'ahc_r': '2642.8', 'ahc_t': '2642.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '88.81', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849664405 ms)
2025-02-17 15:57:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849724414 ms)
2025-02-17 15:57:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849724415 ms)
2025-02-17 15:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:57:25.400188+00:00', 'runtime': '849720.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.518', 'adc_vl_f': '14.59', 'adc_ic_f': '5.974', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.0', 'ahc_t': '2643.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '86.968', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849724416 ms)
2025-02-17 15:57:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:56:25.400179+00:00', 'runtime': '849660.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.581', 'adc_ic_f': '5.989', 'adc_il_f': '0.285', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.603', 'vb_ref': '14.119', 'ahc_r': '2642.9', 'ahc_t': '2642.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '87.149', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '12.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849724416 ms)
2025-02-17 15:58:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849784407 ms)
2025-02-17 15:58:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849784407 ms)
2025-02-17 15:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:58:25.400191+00:00', 'runtime': '849780.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.515', 'adc_vl_f': '14.581', 'adc_ic_f': '5.986', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.1', 'ahc_t': '2643.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '87.406', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849784408 ms)
2025-02-17 15:58:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:57:25.400188+00:00', 'runtime': '849720.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.518', 'adc_vl_f': '14.59', 'adc_ic_f': '5.974', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.0', 'ahc_t': '2643.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '86.968', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849784409 ms)
2025-02-17 15:58:35.471 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+849793290 ms)
2025-02-17 15:58:35.472 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+849793291 ms)
2025-02-17 15:58:35.473 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 15:58:25.400344+00:00'} (T+849793293 ms)
2025-02-17 15:59:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849844418 ms)
2025-02-17 15:59:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849844419 ms)
2025-02-17 15:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 15:59:25.400257+00:00', 'runtime': '849840.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.575', 'adc_ic_f': '5.948', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.2', 'ahc_t': '2643.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '86.907', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849844420 ms)
2025-02-17 15:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:58:25.400191+00:00', 'runtime': '849780.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.515', 'adc_vl_f': '14.581', 'adc_ic_f': '5.986', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.1', 'ahc_t': '2643.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '87.406', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849844420 ms)
2025-02-17 16:00:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849904412 ms)
2025-02-17 16:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849904413 ms)
2025-02-17 16:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:00:25.400150+00:00', 'runtime': '849900.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.575', 'adc_ic_f': '5.909', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.3', 'ahc_t': '2643.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '86.168', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849904414 ms)
2025-02-17 16:00:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 15:59:25.400257+00:00', 'runtime': '849840.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.575', 'adc_ic_f': '5.948', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.2', 'ahc_t': '2643.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '86.907', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849904414 ms)
2025-02-17 16:01:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+849964407 ms)
2025-02-17 16:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+849964407 ms)
2025-02-17 16:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:01:25.400617+00:00', 'runtime': '849960.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.578', 'adc_ic_f': '5.899', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.4', 'ahc_t': '2643.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.775', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849964408 ms)
2025-02-17 16:01:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:00:25.400150+00:00', 'runtime': '849900.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.575', 'adc_ic_f': '5.909', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.3', 'ahc_t': '2643.3', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '86.168', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+849964409 ms)
2025-02-17 16:02:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850024418 ms)
2025-02-17 16:02:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850024418 ms)
2025-02-17 16:02:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:02:25.400212+00:00', 'runtime': '850020.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.597', 'adc_ic_f': '5.849', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.5', 'ahc_t': '2643.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.699', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850024422 ms)
2025-02-17 16:02:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:01:25.400617+00:00', 'runtime': '849960.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.578', 'adc_ic_f': '5.899', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.4', 'ahc_t': '2643.4', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.775', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.4', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850024423 ms)
2025-02-17 16:02:35.479 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850033298 ms)
2025-02-17 16:02:35.480 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850033299 ms)
2025-02-17 16:02:35.480 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:02:25.400334+00:00'} (T+850033300 ms)
2025-02-17 16:03:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850084414 ms)
2025-02-17 16:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850084415 ms)
2025-02-17 16:03:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:03:25.400196+00:00', 'runtime': '850080.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.573', 'adc_vl_f': '14.603', 'adc_ic_f': '5.841', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.6', 'ahc_t': '2643.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.443', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850084419 ms)
2025-02-17 16:03:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:02:25.400212+00:00', 'runtime': '850020.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.597', 'adc_ic_f': '5.849', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.5', 'ahc_t': '2643.5', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.699', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.5', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850084419 ms)
2025-02-17 16:03:35.765 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850093585 ms)
2025-02-17 16:03:35.766 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850093585 ms)
2025-02-17 16:03:35.767 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:03:25.400339+00:00'} (T+850093586 ms)
2025-02-17 16:04:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850144408 ms)
2025-02-17 16:04:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850144409 ms)
2025-02-17 16:04:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:04:25.402626+00:00', 'runtime': '850140.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.686', 'adc_vl_f': '14.606', 'adc_ic_f': '5.781', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2643.7', 'ahc_t': '2643.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.005', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850144413 ms)
2025-02-17 16:04:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:03:25.400196+00:00', 'runtime': '850080.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.573', 'adc_vl_f': '14.603', 'adc_ic_f': '5.841', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.6', 'ahc_t': '2643.6', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.443', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.6', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850144413 ms)
2025-02-17 16:04:35.756 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850153576 ms)
2025-02-17 16:04:35.757 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850153576 ms)
2025-02-17 16:04:35.761 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:04:25.400031+00:00'} (T+850153581 ms)
2025-02-17 16:05:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850204418 ms)
2025-02-17 16:05:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850204418 ms)
2025-02-17 16:05:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:05:25.402636+00:00', 'runtime': '850200.003', 'adc_vb_f': '14.597', 'adc_va_f': '21.671', 'adc_vl_f': '14.606', 'adc_ic_f': '5.742', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2643.8', 'ahc_t': '2643.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '84.567', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850204422 ms)
2025-02-17 16:05:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:04:25.402626+00:00', 'runtime': '850140.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.686', 'adc_vl_f': '14.606', 'adc_ic_f': '5.781', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2643.7', 'ahc_t': '2643.7', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '85.005', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.7', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850204423 ms)
2025-02-17 16:05:35.639 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850213459 ms)
2025-02-17 16:05:35.640 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850213459 ms)
2025-02-17 16:05:35.642 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:05:25.400034+00:00'} (T+850213461 ms)
2025-02-17 16:05:53.629 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+850231449 ms)
2025-02-17 16:05:53.630 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+850231449 ms)
2025-02-17 16:05:53.630 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43030)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+850231449 ms)
2025-02-17 16:06:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850264413 ms)
2025-02-17 16:06:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850264414 ms)
2025-02-17 16:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:06:25.400155+00:00', 'runtime': '850260.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '5.706', 'adc_il_f': '0.273', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.9', 'ahc_t': '2643.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '84.159', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850264415 ms)
2025-02-17 16:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:05:25.402636+00:00', 'runtime': '850200.003', 'adc_vb_f': '14.597', 'adc_va_f': '21.671', 'adc_vl_f': '14.606', 'adc_ic_f': '5.742', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2643.8', 'ahc_t': '2643.8', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '84.567', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.8', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850264415 ms)
2025-02-17 16:06:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850273422 ms)
2025-02-17 16:06:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850273422 ms)
2025-02-17 16:06:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:06:25.400159+00:00'} (T+850273423 ms)
2025-02-17 16:07:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850324408 ms)
2025-02-17 16:07:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850324409 ms)
2025-02-17 16:07:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:07:25.400189+00:00', 'runtime': '850320.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.716', 'adc_vl_f': '14.603', 'adc_ic_f': '5.696', 'adc_il_f': '0.309', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.0', 'ahc_t': '2644.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '83.601', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850324410 ms)
2025-02-17 16:07:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:06:25.400155+00:00', 'runtime': '850260.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '5.706', 'adc_il_f': '0.273', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2643.9', 'ahc_t': '2643.9', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '84.159', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '13.9', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850324410 ms)
2025-02-17 16:07:35.900 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850333720 ms)
2025-02-17 16:07:35.901 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850333720 ms)
2025-02-17 16:07:35.902 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:07:25.400205+00:00'} (T+850333721 ms)
2025-02-17 16:08:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850384404 ms)
2025-02-17 16:08:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850384405 ms)
2025-02-17 16:08:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:08:25.400197+00:00', 'runtime': '850380.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '5.696', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.1', 'ahc_t': '2644.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '83.465', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850384405 ms)
2025-02-17 16:08:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:07:25.400189+00:00', 'runtime': '850320.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.716', 'adc_vl_f': '14.603', 'adc_ic_f': '5.696', 'adc_il_f': '0.309', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.0', 'ahc_t': '2644.0', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '83.601', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.0', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850384406 ms)
2025-02-17 16:08:36.446 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850394265 ms)
2025-02-17 16:08:36.447 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850394266 ms)
2025-02-17 16:08:36.448 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:08:25.400433+00:00'} (T+850394267 ms)
2025-02-17 16:09:05.938 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+850423757 ms)
2025-02-17 16:09:05.938 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+850423757 ms)
2025-02-17 16:09:05.939 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43236)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+850423758 ms)
2025-02-17 16:09:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850444416 ms)
2025-02-17 16:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850444417 ms)
2025-02-17 16:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:09:25.400162+00:00', 'runtime': '850440.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.664', 'adc_vl_f': '14.603', 'adc_ic_f': '5.663', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.2', 'ahc_t': '2644.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '82.65', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850444418 ms)
2025-02-17 16:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:08:25.400197+00:00', 'runtime': '850380.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '5.696', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.1', 'ahc_t': '2644.1', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '83.465', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.1', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850444418 ms)
2025-02-17 16:10:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850504413 ms)
2025-02-17 16:10:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850504413 ms)
2025-02-17 16:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:10:25.400427+00:00', 'runtime': '850500.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.606', 'adc_vl_f': '14.603', 'adc_ic_f': '5.634', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.3', 'ahc_t': '2644.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.864', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850504416 ms)
2025-02-17 16:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:09:25.400162+00:00', 'runtime': '850440.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.664', 'adc_vl_f': '14.603', 'adc_ic_f': '5.663', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.2', 'ahc_t': '2644.2', 'kwhc': '34.2', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '82.65', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.2', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850504416 ms)
2025-02-17 16:10:30.118 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+850507938 ms)
2025-02-17 16:10:30.119 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+850507938 ms)
2025-02-17 16:10:30.119 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43326)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+850507938 ms)
2025-02-17 16:11:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850564408 ms)
2025-02-17 16:11:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850564408 ms)
2025-02-17 16:11:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:11:25.400219+00:00', 'runtime': '850560.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.677', 'adc_vl_f': '14.603', 'adc_ic_f': '5.597', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.4', 'ahc_t': '2644.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.562', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850564410 ms)
2025-02-17 16:11:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:10:25.400427+00:00', 'runtime': '850500.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.606', 'adc_vl_f': '14.603', 'adc_ic_f': '5.634', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.3', 'ahc_t': '2644.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.0', 'ahl_t': '1591.0', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.864', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.3', 'ahl_daily': '6.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850564410 ms)
2025-02-17 16:12:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850624403 ms)
2025-02-17 16:12:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850624404 ms)
2025-02-17 16:12:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:12:25.400192+00:00', 'runtime': '850620.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '5.559', 'adc_il_f': '0.297', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2644.5', 'ahc_t': '2644.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.653', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850624406 ms)
2025-02-17 16:12:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:11:25.400219+00:00', 'runtime': '850560.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.677', 'adc_vl_f': '14.603', 'adc_ic_f': '5.597', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.4', 'ahc_t': '2644.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.562', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850624406 ms)
2025-02-17 16:13:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850684415 ms)
2025-02-17 16:13:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850684415 ms)
2025-02-17 16:13:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:13:25.400209+00:00', 'runtime': '850680.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.68', 'adc_vl_f': '14.603', 'adc_ic_f': '5.501', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.6', 'ahc_t': '2644.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.019', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850684417 ms)
2025-02-17 16:13:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:12:25.400192+00:00', 'runtime': '850620.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '5.559', 'adc_il_f': '0.297', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2644.5', 'ahc_t': '2644.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.653', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850684418 ms)
2025-02-17 16:14:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850744409 ms)
2025-02-17 16:14:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850744410 ms)
2025-02-17 16:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:14:25.400183+00:00', 'runtime': '850740.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.664', 'adc_vl_f': '14.603', 'adc_ic_f': '5.474', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2644.6', 'ahc_t': '2644.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '80.4', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850744412 ms)
2025-02-17 16:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:13:25.400209+00:00', 'runtime': '850680.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.68', 'adc_vl_f': '14.603', 'adc_ic_f': '5.501', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.6', 'ahc_t': '2644.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '81.019', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850744412 ms)
2025-02-17 16:15:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850804406 ms)
2025-02-17 16:15:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850804407 ms)
2025-02-17 16:15:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:15:25.400170+00:00', 'runtime': '850800.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.597', 'adc_ic_f': '5.428', 'adc_il_f': '0.287', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.7', 'ahc_t': '2644.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '79.117', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.7', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850804408 ms)
2025-02-17 16:15:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:14:25.400183+00:00', 'runtime': '850740.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.664', 'adc_vl_f': '14.603', 'adc_ic_f': '5.474', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2644.6', 'ahc_t': '2644.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '80.4', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850804408 ms)
2025-02-17 16:16:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850864417 ms)
2025-02-17 16:16:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850864417 ms)
2025-02-17 16:16:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:16:25.400175+00:00', 'runtime': '850860.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.587', 'adc_ic_f': '5.428', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2644.8', 'ahc_t': '2644.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '79.509', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850864418 ms)
2025-02-17 16:16:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:15:25.400170+00:00', 'runtime': '850800.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.597', 'adc_ic_f': '5.428', 'adc_il_f': '0.287', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.7', 'ahc_t': '2644.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '79.117', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.7', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850864419 ms)
2025-02-17 16:16:27.635 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+850865454 ms)
2025-02-17 16:16:27.635 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+850865454 ms)
2025-02-17 16:16:27.636 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43710)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+850865455 ms)
2025-02-17 16:16:35.926 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850873745 ms)
2025-02-17 16:16:35.927 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850873746 ms)
2025-02-17 16:16:35.932 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:16:25.400340+00:00'} (T+850873751 ms)
2025-02-17 16:17:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850924413 ms)
2025-02-17 16:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850924413 ms)
2025-02-17 16:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:17:25.400175+00:00', 'runtime': '850920.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.597', 'adc_ic_f': '5.368', 'adc_il_f': '0.263', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.9', 'ahc_t': '2644.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '79.117', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.9', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850924415 ms)
2025-02-17 16:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:16:25.400175+00:00', 'runtime': '850860.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.587', 'adc_ic_f': '5.428', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2644.8', 'ahc_t': '2644.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '79.509', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850924415 ms)
2025-02-17 16:17:35.658 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850933477 ms)
2025-02-17 16:17:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850933478 ms)
2025-02-17 16:17:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:17:25.400380+00:00'} (T+850933479 ms)
2025-02-17 16:18:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+850984424 ms)
2025-02-17 16:18:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+850984425 ms)
2025-02-17 16:18:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:18:25.400546+00:00', 'runtime': '850980.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.594', 'adc_ic_f': '5.361', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.0', 'ahc_t': '2645.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '78.633', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.0', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850984426 ms)
2025-02-17 16:18:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:17:25.400175+00:00', 'runtime': '850920.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.597', 'adc_ic_f': '5.368', 'adc_il_f': '0.263', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2644.9', 'ahc_t': '2644.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '79.117', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '14.9', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+850984426 ms)
2025-02-17 16:18:35.976 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850993795 ms)
2025-02-17 16:18:35.976 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+850993795 ms)
2025-02-17 16:18:35.978 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:18:25.400168+00:00'} (T+850993797 ms)
2025-02-17 16:19:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851044420 ms)
2025-02-17 16:19:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851044420 ms)
2025-02-17 16:19:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:19:25.400184+00:00', 'runtime': '851040.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.59', 'adc_ic_f': '5.341', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.1', 'ahc_t': '2645.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '78.18', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.1', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851044421 ms)
2025-02-17 16:19:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:18:25.400546+00:00', 'runtime': '850980.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.594', 'adc_ic_f': '5.361', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.0', 'ahc_t': '2645.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '78.633', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.0', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851044422 ms)
2025-02-17 16:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851104413 ms)
2025-02-17 16:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851104413 ms)
2025-02-17 16:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:20:25.400775+00:00', 'runtime': '851100.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.545', 'adc_vl_f': '14.584', 'adc_ic_f': '5.312', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.2', 'ahc_t': '2645.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '77.093', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.2', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851104414 ms)
2025-02-17 16:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:19:25.400184+00:00', 'runtime': '851040.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.59', 'adc_ic_f': '5.341', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.1', 'ahc_t': '2645.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '78.18', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.1', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851104415 ms)
2025-02-17 16:21:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851164411 ms)
2025-02-17 16:21:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851164411 ms)
2025-02-17 16:21:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:21:25.400627+00:00', 'runtime': '851160.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.457', 'adc_vl_f': '14.575', 'adc_ic_f': '5.288', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.3', 'ahc_t': '2645.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '77.41', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.3', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851164412 ms)
2025-02-17 16:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:20:25.400775+00:00', 'runtime': '851100.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.545', 'adc_vl_f': '14.584', 'adc_ic_f': '5.312', 'adc_il_f': '0.3', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.2', 'ahc_t': '2645.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '77.093', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.2', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851164413 ms)
2025-02-17 16:22:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851224406 ms)
2025-02-17 16:22:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851224407 ms)
2025-02-17 16:22:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:22:25.402816+00:00', 'runtime': '851220.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.569', 'adc_ic_f': '5.257', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.4', 'ahc_t': '2645.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '76.912', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851224408 ms)
2025-02-17 16:22:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:21:25.400627+00:00', 'runtime': '851160.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.457', 'adc_vl_f': '14.575', 'adc_ic_f': '5.288', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.3', 'ahc_t': '2645.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '77.41', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.3', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851224409 ms)
2025-02-17 16:22:35.751 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851233570 ms)
2025-02-17 16:22:35.752 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851233571 ms)
2025-02-17 16:22:35.753 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:22:25.400199+00:00'} (T+851233572 ms)
2025-02-17 16:22:46.105 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+851243925 ms)
2025-02-17 16:22:46.106 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+851243925 ms)
2025-02-17 16:22:46.106 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+851243925 ms)
2025-02-17 16:22:46.107 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+851243926 ms)
2025-02-17 16:22:46.107 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+851243926 ms)
2025-02-17 16:22:46.107 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+851243927 ms)
2025-02-17 16:23:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851284419 ms)
2025-02-17 16:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851284419 ms)
2025-02-17 16:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:23:25.402611+00:00', 'runtime': '851280.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.506', 'adc_vl_f': '14.581', 'adc_ic_f': '5.223', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.4', 'ahc_t': '2645.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '76.399', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851284420 ms)
2025-02-17 16:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:22:25.402816+00:00', 'runtime': '851220.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.451', 'adc_vl_f': '14.569', 'adc_ic_f': '5.257', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.4', 'ahc_t': '2645.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '76.912', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851284420 ms)
2025-02-17 16:23:35.481 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851293300 ms)
2025-02-17 16:23:35.482 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851293301 ms)
2025-02-17 16:23:35.484 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:23:25.400034+00:00'} (T+851293303 ms)
2025-02-17 16:24:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851344415 ms)
2025-02-17 16:24:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851344415 ms)
2025-02-17 16:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:24:25.402802+00:00', 'runtime': '851340.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.59', 'adc_ic_f': '5.189', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.5', 'ahc_t': '2645.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '75.916', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851344416 ms)
2025-02-17 16:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:23:25.402611+00:00', 'runtime': '851280.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.506', 'adc_vl_f': '14.581', 'adc_ic_f': '5.223', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.4', 'ahc_t': '2645.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '76.399', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851344416 ms)
2025-02-17 16:25:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851404412 ms)
2025-02-17 16:25:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851404413 ms)
2025-02-17 16:25:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:25:25.400865+00:00', 'runtime': '851400.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.548', 'adc_vl_f': '14.584', 'adc_ic_f': '5.153', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.6', 'ahc_t': '2645.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '75.523', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851404414 ms)
2025-02-17 16:25:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:24:25.402802+00:00', 'runtime': '851340.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.59', 'adc_ic_f': '5.189', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.5', 'ahc_t': '2645.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '75.916', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851404414 ms)
2025-02-17 16:26:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851464407 ms)
2025-02-17 16:26:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851464407 ms)
2025-02-17 16:26:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:26:25.400579+00:00', 'runtime': '851460.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.59', 'adc_ic_f': '5.112', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2645.7', 'ahc_t': '2645.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '75.055', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.7', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851464408 ms)
2025-02-17 16:26:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:25:25.400865+00:00', 'runtime': '851400.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.548', 'adc_vl_f': '14.584', 'adc_ic_f': '5.153', 'adc_il_f': '0.304', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2645.6', 'ahc_t': '2645.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '75.523', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851464409 ms)
2025-02-17 16:27:25.649 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851523469 ms)
2025-02-17 16:27:25.650 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851523469 ms)
2025-02-17 16:27:25.651 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:27:25.402781+00:00', 'runtime': '851520.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '5.088', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2645.8', 'ahc_t': '2645.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '74.149', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851523470 ms)
2025-02-17 16:27:25.651 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:26:25.400579+00:00', 'runtime': '851460.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.59', 'adc_ic_f': '5.112', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2645.7', 'ahc_t': '2645.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '75.055', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.7', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851523470 ms)
2025-02-17 16:27:35.850 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851533669 ms)
2025-02-17 16:27:35.851 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851533670 ms)
2025-02-17 16:27:35.852 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:27:25.400122+00:00'} (T+851533671 ms)
2025-02-17 16:27:42.540 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+851540359 ms)
2025-02-17 16:27:42.541 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+851540360 ms)
2025-02-17 16:27:42.541 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44434)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+851540360 ms)
2025-02-17 16:28:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851584416 ms)
2025-02-17 16:28:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851584417 ms)
2025-02-17 16:28:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:28:25.402744+00:00', 'runtime': '851580.003', 'adc_vb_f': '14.56', 'adc_va_f': '21.725', 'adc_vl_f': '14.532', 'adc_ic_f': '4.873', 'adc_il_f': '0.271', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2645.9', 'ahc_t': '2645.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '73.364', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.9', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851584418 ms)
2025-02-17 16:28:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:27:25.402781+00:00', 'runtime': '851520.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '5.088', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2645.8', 'ahc_t': '2645.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '74.149', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851584418 ms)
2025-02-17 16:28:35.983 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851593802 ms)
2025-02-17 16:28:35.984 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851593803 ms)
2025-02-17 16:28:35.985 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:28:25.400154+00:00'} (T+851593804 ms)
2025-02-17 16:29:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851644412 ms)
2025-02-17 16:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851644412 ms)
2025-02-17 16:29:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:29:25.402779+00:00', 'runtime': '851640.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.686', 'adc_vl_f': '14.563', 'adc_ic_f': '5.03', 'adc_il_f': '0.275', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.0', 'ahc_t': '2646.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '73.334', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.0', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851644413 ms)
2025-02-17 16:29:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:28:25.402744+00:00', 'runtime': '851580.003', 'adc_vb_f': '14.56', 'adc_va_f': '21.725', 'adc_vl_f': '14.532', 'adc_ic_f': '4.873', 'adc_il_f': '0.271', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2645.9', 'ahc_t': '2645.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '73.364', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '15.9', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851644414 ms)
2025-02-17 16:30:20.740 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+851698559 ms)
2025-02-17 16:30:20.740 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+851698560 ms)
2025-02-17 16:30:20.741 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+851698560 ms)
2025-02-17 16:30:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851704411 ms)
2025-02-17 16:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851704411 ms)
2025-02-17 16:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:30:25.400184+00:00', 'runtime': '851700.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.667', 'adc_vl_f': '14.569', 'adc_ic_f': '4.979', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.0', 'ahc_t': '2646.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '72.518', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.0', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851704412 ms)
2025-02-17 16:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:29:25.402779+00:00', 'runtime': '851640.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.686', 'adc_vl_f': '14.563', 'adc_ic_f': '5.03', 'adc_il_f': '0.275', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.0', 'ahc_t': '2646.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '73.334', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.0', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851704412 ms)
2025-02-17 16:31:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851764406 ms)
2025-02-17 16:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851764406 ms)
2025-02-17 16:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:31:25.400171+00:00', 'runtime': '851760.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.765', 'adc_vl_f': '14.569', 'adc_ic_f': '4.95', 'adc_il_f': '0.312', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.1', 'ahc_t': '2646.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '72.141', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.1', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851764407 ms)
2025-02-17 16:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:30:25.400184+00:00', 'runtime': '851700.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.667', 'adc_vl_f': '14.569', 'adc_ic_f': '4.979', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.0', 'ahc_t': '2646.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '72.518', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.0', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851764408 ms)
2025-02-17 16:32:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851824417 ms)
2025-02-17 16:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851824418 ms)
2025-02-17 16:32:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:32:25.400173+00:00', 'runtime': '851820.001', 'adc_vb_f': '14.566', 'adc_va_f': '21.863', 'adc_vl_f': '14.542', 'adc_ic_f': '4.856', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.2', 'ahc_t': '2646.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '71.522', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.2', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851824419 ms)
2025-02-17 16:32:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:31:25.400171+00:00', 'runtime': '851760.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.765', 'adc_vl_f': '14.569', 'adc_ic_f': '4.95', 'adc_il_f': '0.312', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.1', 'ahc_t': '2646.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '72.141', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.1', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851824419 ms)
2025-02-17 16:33:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851884415 ms)
2025-02-17 16:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851884416 ms)
2025-02-17 16:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:33:25.400164+00:00', 'runtime': '851880.001', 'adc_vb_f': '14.572', 'adc_va_f': '21.814', 'adc_vl_f': '14.554', 'adc_ic_f': '4.909', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.3', 'ahc_t': '2646.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '71.613', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.3', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851884417 ms)
2025-02-17 16:33:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:32:25.400173+00:00', 'runtime': '851820.001', 'adc_vb_f': '14.566', 'adc_va_f': '21.863', 'adc_vl_f': '14.542', 'adc_ic_f': '4.856', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.2', 'ahc_t': '2646.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '71.522', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.2', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851884417 ms)
2025-02-17 16:34:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+851944411 ms)
2025-02-17 16:34:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+851944411 ms)
2025-02-17 16:34:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:34:25.400177+00:00', 'runtime': '851940.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.774', 'adc_vl_f': '14.569', 'adc_ic_f': '4.851', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.4', 'ahc_t': '2646.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '70.692', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851944412 ms)
2025-02-17 16:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:33:25.400164+00:00', 'runtime': '851880.001', 'adc_vb_f': '14.572', 'adc_va_f': '21.814', 'adc_vl_f': '14.554', 'adc_ic_f': '4.909', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.3', 'ahc_t': '2646.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '71.613', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.3', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+851944413 ms)
2025-02-17 16:34:35.823 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851953643 ms)
2025-02-17 16:34:35.824 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+851953643 ms)
2025-02-17 16:34:35.826 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:34:25.400345+00:00'} (T+851953645 ms)
2025-02-17 16:35:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852004407 ms)
2025-02-17 16:35:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852004407 ms)
2025-02-17 16:35:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:35:25.400177+00:00', 'runtime': '852000.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.569', 'adc_ic_f': '4.834', 'adc_il_f': '0.343', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.5', 'ahc_t': '2646.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '70.42', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852004408 ms)
2025-02-17 16:35:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:34:25.400177+00:00', 'runtime': '851940.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.774', 'adc_vl_f': '14.569', 'adc_ic_f': '4.851', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.4', 'ahc_t': '2646.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '70.692', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.4', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852004409 ms)
2025-02-17 16:35:35.912 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852013731 ms)
2025-02-17 16:35:35.913 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852013732 ms)
2025-02-17 16:35:35.916 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:35:25.400009+00:00'} (T+852013735 ms)
2025-02-17 16:36:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852064418 ms)
2025-02-17 16:36:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852064418 ms)
2025-02-17 16:36:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:36:25.400171+00:00', 'runtime': '852060.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.78', 'adc_vl_f': '14.569', 'adc_ic_f': '4.822', 'adc_il_f': '0.333', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.5', 'ahc_t': '2646.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '70.269', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852064419 ms)
2025-02-17 16:36:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:35:25.400177+00:00', 'runtime': '852000.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.569', 'adc_ic_f': '4.834', 'adc_il_f': '0.343', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.5', 'ahc_t': '2646.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '70.42', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852064420 ms)
2025-02-17 16:36:35.882 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852073701 ms)
2025-02-17 16:36:35.883 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852073702 ms)
2025-02-17 16:36:35.888 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:36:25.400412+00:00'} (T+852073707 ms)
2025-02-17 16:37:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852124414 ms)
2025-02-17 16:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852124414 ms)
2025-02-17 16:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:37:25.400169+00:00', 'runtime': '852120.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.878', 'adc_vl_f': '14.569', 'adc_ic_f': '4.774', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.6', 'ahc_t': '2646.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '69.529', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852124415 ms)
2025-02-17 16:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:36:25.400171+00:00', 'runtime': '852060.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.78', 'adc_vl_f': '14.569', 'adc_ic_f': '4.822', 'adc_il_f': '0.333', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.5', 'ahc_t': '2646.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '70.269', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.5', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852124416 ms)
2025-02-17 16:37:35.772 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852133592 ms)
2025-02-17 16:37:35.773 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852133592 ms)
2025-02-17 16:37:35.777 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:37:25.400450+00:00'} (T+852133597 ms)
2025-02-17 16:38:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852184409 ms)
2025-02-17 16:38:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852184410 ms)
2025-02-17 16:38:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:38:25.402572+00:00', 'runtime': '852180.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.869', 'adc_vl_f': '14.569', 'adc_ic_f': '4.764', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.7', 'ahc_t': '2646.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '69.393', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.7', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852184411 ms)
2025-02-17 16:38:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:37:25.400169+00:00', 'runtime': '852120.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.878', 'adc_vl_f': '14.569', 'adc_ic_f': '4.774', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.6', 'ahc_t': '2646.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '69.529', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.6', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852184411 ms)
2025-02-17 16:38:35.761 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852193580 ms)
2025-02-17 16:38:35.762 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852193581 ms)
2025-02-17 16:38:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:38:25.400016+00:00'} (T+852193582 ms)
2025-02-17 16:38:48.461 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+852206280 ms)
2025-02-17 16:38:48.461 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+852206281 ms)
2025-02-17 16:38:48.462 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45148)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+852206281 ms)
2025-02-17 16:39:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852244407 ms)
2025-02-17 16:39:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852244407 ms)
2025-02-17 16:39:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:39:25.400026+00:00', 'runtime': '852240.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.569', 'adc_ic_f': '4.713', 'adc_il_f': '0.263', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.8', 'ahc_t': '2646.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '68.653', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852244408 ms)
2025-02-17 16:39:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:38:25.402572+00:00', 'runtime': '852180.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.869', 'adc_vl_f': '14.569', 'adc_ic_f': '4.764', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.7', 'ahc_t': '2646.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '69.393', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.7', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852244409 ms)
2025-02-17 16:39:35.447 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852253266 ms)
2025-02-17 16:39:35.447 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852253266 ms)
2025-02-17 16:39:35.448 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:39:25.405126+00:00'} (T+852253268 ms)
2025-02-17 16:40:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852304420 ms)
2025-02-17 16:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852304421 ms)
2025-02-17 16:40:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:40:25.400207+00:00', 'runtime': '852300.001', 'adc_vb_f': '14.569', 'adc_va_f': '21.832', 'adc_vl_f': '14.548', 'adc_ic_f': '4.36', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.8', 'ahc_t': '2646.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '64.411', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852304422 ms)
2025-02-17 16:40:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:39:25.400026+00:00', 'runtime': '852240.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.569', 'adc_ic_f': '4.713', 'adc_il_f': '0.263', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2646.8', 'ahc_t': '2646.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '68.653', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852304422 ms)
2025-02-17 16:40:36.703 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852314522 ms)
2025-02-17 16:40:36.703 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852314523 ms)
2025-02-17 16:40:36.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:40:25.413127+00:00'} (T+852314525 ms)
2025-02-17 16:41:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852364418 ms)
2025-02-17 16:41:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852364419 ms)
2025-02-17 16:41:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:41:25.402684+00:00', 'runtime': '852360.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.566', 'adc_ic_f': '4.643', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.9', 'ahc_t': '2646.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '67.702', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.9', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852364420 ms)
2025-02-17 16:41:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:40:25.400207+00:00', 'runtime': '852300.001', 'adc_vb_f': '14.569', 'adc_va_f': '21.832', 'adc_vl_f': '14.548', 'adc_ic_f': '4.36', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.8', 'ahc_t': '2646.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.1', 'ahl_t': '1591.1', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '64.411', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.8', 'ahl_daily': '6.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852364420 ms)
2025-02-17 16:41:35.908 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852373728 ms)
2025-02-17 16:41:35.909 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852373728 ms)
2025-02-17 16:41:35.911 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:41:25.400075+00:00'} (T+852373730 ms)
2025-02-17 16:41:55.774 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+852393593 ms)
2025-02-17 16:41:55.774 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+852393593 ms)
2025-02-17 16:41:55.775 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45350)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+852393594 ms)
2025-02-17 16:42:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852424416 ms)
2025-02-17 16:42:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852424416 ms)
2025-02-17 16:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:42:25.402783+00:00', 'runtime': '852420.003', 'adc_vb_f': '14.566', 'adc_va_f': '21.777', 'adc_vl_f': '14.532', 'adc_ic_f': '4.575', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.0', 'ahc_t': '2647.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '67.325', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.0', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852424417 ms)
2025-02-17 16:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:41:25.402684+00:00', 'runtime': '852360.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.566', 'adc_ic_f': '4.643', 'adc_il_f': '0.278', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2646.9', 'ahc_t': '2646.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '67.702', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '16.9', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852424418 ms)
2025-02-17 16:42:37.024 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852434843 ms)
2025-02-17 16:42:37.024 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852434843 ms)
2025-02-17 16:42:37.025 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:42:25.400216+00:00'} (T+852434844 ms)
2025-02-17 16:43:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852484411 ms)
2025-02-17 16:43:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852484412 ms)
2025-02-17 16:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:43:25.400160+00:00', 'runtime': '852480.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.569', 'adc_ic_f': '4.571', 'adc_il_f': '0.309', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.1', 'ahc_t': '2647.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '66.57', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.1', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852484414 ms)
2025-02-17 16:43:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:42:25.402783+00:00', 'runtime': '852420.003', 'adc_vb_f': '14.566', 'adc_va_f': '21.777', 'adc_vl_f': '14.532', 'adc_ic_f': '4.575', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.0', 'ahc_t': '2647.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '67.325', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.0', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852484414 ms)
2025-02-17 16:44:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852544408 ms)
2025-02-17 16:44:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852544408 ms)
2025-02-17 16:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:44:25.400198+00:00', 'runtime': '852540.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.683', 'adc_vl_f': '14.569', 'adc_ic_f': '4.534', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.2', 'ahc_t': '2647.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '66.041', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852544410 ms)
2025-02-17 16:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:43:25.400160+00:00', 'runtime': '852480.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.771', 'adc_vl_f': '14.569', 'adc_ic_f': '4.571', 'adc_il_f': '0.309', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.1', 'ahc_t': '2647.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '66.57', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.1', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852544410 ms)
2025-02-17 16:45:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852604406 ms)
2025-02-17 16:45:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852604406 ms)
2025-02-17 16:45:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:45:25.400257+00:00', 'runtime': '852600.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.68', 'adc_vl_f': '14.569', 'adc_ic_f': '4.52', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.2', 'ahc_t': '2647.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '65.83', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852604408 ms)
2025-02-17 16:45:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:44:25.400198+00:00', 'runtime': '852540.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.683', 'adc_vl_f': '14.569', 'adc_ic_f': '4.534', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.2', 'ahc_t': '2647.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '66.041', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852604409 ms)
2025-02-17 16:45:35.819 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852613638 ms)
2025-02-17 16:45:35.820 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852613639 ms)
2025-02-17 16:45:35.825 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:45:25.400351+00:00'} (T+852613644 ms)
2025-02-17 16:46:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852664416 ms)
2025-02-17 16:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852664417 ms)
2025-02-17 16:46:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:46:25.400204+00:00', 'runtime': '852660.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.661', 'adc_vl_f': '14.569', 'adc_ic_f': '4.474', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.3', 'ahc_t': '2647.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '65.196', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.3', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852664418 ms)
2025-02-17 16:46:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:45:25.400257+00:00', 'runtime': '852600.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.68', 'adc_vl_f': '14.569', 'adc_ic_f': '4.52', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.2', 'ahc_t': '2647.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '65.83', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852664418 ms)
2025-02-17 16:46:35.639 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852673458 ms)
2025-02-17 16:46:35.639 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852673459 ms)
2025-02-17 16:46:35.641 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:46:25.399984+00:00'} (T+852673460 ms)
2025-02-17 16:47:20.264 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+852718083 ms)
2025-02-17 16:47:20.264 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+852718083 ms)
2025-02-17 16:47:20.264 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45704)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+852718084 ms)
2025-02-17 16:47:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852724415 ms)
2025-02-17 16:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852724415 ms)
2025-02-17 16:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:47:25.400161+00:00', 'runtime': '852720.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.498', 'adc_il_f': '0.312', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.4', 'ahc_t': '2647.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '65.558', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.4', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852724416 ms)
2025-02-17 16:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:46:25.400204+00:00', 'runtime': '852660.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.661', 'adc_vl_f': '14.569', 'adc_ic_f': '4.474', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.3', 'ahc_t': '2647.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '65.196', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.3', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852724417 ms)
2025-02-17 16:47:35.679 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852733498 ms)
2025-02-17 16:47:35.680 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852733499 ms)
2025-02-17 16:47:35.685 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:47:25.400149+00:00'} (T+852733504 ms)
2025-02-17 16:48:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852784409 ms)
2025-02-17 16:48:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852784410 ms)
2025-02-17 16:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:48:25.400171+00:00', 'runtime': '852780.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.418', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.5', 'ahc_t': '2647.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '64.395', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.5', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852784414 ms)
2025-02-17 16:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:47:25.400161+00:00', 'runtime': '852720.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.498', 'adc_il_f': '0.312', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.4', 'ahc_t': '2647.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '65.558', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.4', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852784414 ms)
2025-02-17 16:48:35.759 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852793578 ms)
2025-02-17 16:48:35.760 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852793579 ms)
2025-02-17 16:48:35.761 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:48:25.400115+00:00'} (T+852793580 ms)
2025-02-17 16:49:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852844421 ms)
2025-02-17 16:49:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852844421 ms)
2025-02-17 16:49:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:49:25.400032+00:00', 'runtime': '852840.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.411', 'adc_il_f': '0.287', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.5', 'ahc_t': '2647.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '64.244', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.5', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852844426 ms)
2025-02-17 16:49:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:48:25.400171+00:00', 'runtime': '852780.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.418', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.5', 'ahc_t': '2647.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '64.395', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.5', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852844426 ms)
2025-02-17 16:49:35.829 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852853649 ms)
2025-02-17 16:49:35.830 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+852853649 ms)
2025-02-17 16:49:35.831 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:49:25.405128+00:00'} (T+852853650 ms)
2025-02-17 16:49:41.535 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+852859355 ms)
2025-02-17 16:49:41.536 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+852859355 ms)
2025-02-17 16:49:41.536 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45856)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+852859355 ms)
2025-02-17 16:50:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852904418 ms)
2025-02-17 16:50:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852904418 ms)
2025-02-17 16:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:50:25.402618+00:00', 'runtime': '852900.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '4.356', 'adc_il_f': '0.287', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.6', 'ahc_t': '2647.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '63.474', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.6', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852904419 ms)
2025-02-17 16:50:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:49:25.400032+00:00', 'runtime': '852840.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.411', 'adc_il_f': '0.287', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.5', 'ahc_t': '2647.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '64.244', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.5', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852904420 ms)
2025-02-17 16:50:49.663 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+852927483 ms)
2025-02-17 16:50:49.664 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+852927483 ms)
2025-02-17 16:50:49.664 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45932)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+852927483 ms)
2025-02-17 16:51:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+852964416 ms)
2025-02-17 16:51:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+852964416 ms)
2025-02-17 16:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:51:25.400698+00:00', 'runtime': '852960.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.692', 'adc_vl_f': '14.563', 'adc_ic_f': '4.353', 'adc_il_f': '0.261', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.7', 'ahc_t': '2647.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '63.399', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.7', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852964417 ms)
2025-02-17 16:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:50:25.402618+00:00', 'runtime': '852900.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '4.356', 'adc_il_f': '0.287', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.6', 'ahc_t': '2647.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '63.474', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.6', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+852964418 ms)
2025-02-17 16:52:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853024412 ms)
2025-02-17 16:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853024413 ms)
2025-02-17 16:52:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:52:25.402761+00:00', 'runtime': '853020.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.683', 'adc_vl_f': '14.569', 'adc_ic_f': '4.305', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.7', 'ahc_t': '2647.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '62.704', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.7', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853024414 ms)
2025-02-17 16:52:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:51:25.400698+00:00', 'runtime': '852960.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.692', 'adc_vl_f': '14.563', 'adc_ic_f': '4.353', 'adc_il_f': '0.261', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.7', 'ahc_t': '2647.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '63.399', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.7', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853024414 ms)
2025-02-17 16:52:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853033425 ms)
2025-02-17 16:52:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853033425 ms)
2025-02-17 16:52:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:52:25.400118+00:00'} (T+853033427 ms)
2025-02-17 16:52:49.864 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+853047683 ms)
2025-02-17 16:52:49.864 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+853047684 ms)
2025-02-17 16:52:49.865 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46060)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+853047684 ms)
2025-02-17 16:53:03.953 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+853061772 ms)
2025-02-17 16:53:03.953 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+853061772 ms)
2025-02-17 16:53:03.954 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46076)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+853061773 ms)
2025-02-17 16:53:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853084408 ms)
2025-02-17 16:53:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853084409 ms)
2025-02-17 16:53:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:53:25.400246+00:00', 'runtime': '853080.001', 'adc_vb_f': '14.563', 'adc_va_f': '21.655', 'adc_vl_f': '14.532', 'adc_ic_f': '4.143', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.8', 'ahc_t': '2647.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '62.206', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.8', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853084410 ms)
2025-02-17 16:53:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:52:25.402761+00:00', 'runtime': '853020.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.683', 'adc_vl_f': '14.569', 'adc_ic_f': '4.305', 'adc_il_f': '0.295', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2647.7', 'ahc_t': '2647.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '62.704', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.7', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853084410 ms)
2025-02-17 16:53:35.785 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853093604 ms)
2025-02-17 16:53:35.786 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853093605 ms)
2025-02-17 16:53:35.787 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:53:25.405139+00:00'} (T+853093606 ms)
2025-02-17 16:54:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853144406 ms)
2025-02-17 16:54:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853144406 ms)
2025-02-17 16:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:54:25.400008+00:00', 'runtime': '853140.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.264', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.9', 'ahc_t': '2647.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '62.146', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.9', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853144407 ms)
2025-02-17 16:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:53:25.400246+00:00', 'runtime': '853080.001', 'adc_vb_f': '14.563', 'adc_va_f': '21.655', 'adc_vl_f': '14.532', 'adc_ic_f': '4.143', 'adc_il_f': '0.307', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.8', 'ahc_t': '2647.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20123', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '62.206', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.8', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853144407 ms)
2025-02-17 16:54:35.627 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853153446 ms)
2025-02-17 16:54:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853153446 ms)
2025-02-17 16:54:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:54:25.405136+00:00'} (T+853153447 ms)
2025-02-17 16:54:43.175 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+853160995 ms)
2025-02-17 16:54:43.175 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+853160995 ms)
2025-02-17 16:54:43.176 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46182)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+853160995 ms)
2025-02-17 16:55:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853204416 ms)
2025-02-17 16:55:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853204417 ms)
2025-02-17 16:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:55:25.400212+00:00', 'runtime': '853200.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.619', 'adc_vl_f': '14.569', 'adc_ic_f': '4.216', 'adc_il_f': '0.338', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.0', 'ahc_t': '2648.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '61.436', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.0', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853204418 ms)
2025-02-17 16:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:54:25.400008+00:00', 'runtime': '853140.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.569', 'adc_ic_f': '4.264', 'adc_il_f': '0.302', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2647.9', 'ahc_t': '2647.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '62.146', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '17.9', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853204418 ms)
2025-02-17 16:55:35.709 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853213529 ms)
2025-02-17 16:55:35.710 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853213529 ms)
2025-02-17 16:55:35.711 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 16:55:25.400214+00:00'} (T+853213530 ms)
2025-02-17 16:56:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853264414 ms)
2025-02-17 16:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853264414 ms)
2025-02-17 16:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:56:25.400192+00:00', 'runtime': '853260.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.683', 'adc_vl_f': '14.569', 'adc_ic_f': '4.187', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.0', 'ahc_t': '2648.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '61.013', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.0', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853264415 ms)
2025-02-17 16:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:55:25.400212+00:00', 'runtime': '853200.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.619', 'adc_vl_f': '14.569', 'adc_ic_f': '4.216', 'adc_il_f': '0.338', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.0', 'ahc_t': '2648.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '61.436', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.0', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853264416 ms)
2025-02-17 16:57:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853324408 ms)
2025-02-17 16:57:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853324408 ms)
2025-02-17 16:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:57:25.400184+00:00', 'runtime': '853320.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '4.155', 'adc_il_f': '0.285', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.1', 'ahc_t': '2648.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '60.56', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.1', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853324410 ms)
2025-02-17 16:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:56:25.400192+00:00', 'runtime': '853260.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.683', 'adc_vl_f': '14.569', 'adc_ic_f': '4.187', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.0', 'ahc_t': '2648.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '61.013', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.0', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853324410 ms)
2025-02-17 16:58:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853384405 ms)
2025-02-17 16:58:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853384405 ms)
2025-02-17 16:58:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:58:25.400266+00:00', 'runtime': '853380.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.735', 'adc_vl_f': '14.563', 'adc_ic_f': '3.969', 'adc_il_f': '0.309', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.2', 'ahc_t': '2648.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.159', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853384406 ms)
2025-02-17 16:58:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:57:25.400184+00:00', 'runtime': '853320.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '4.155', 'adc_il_f': '0.285', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.1', 'ahc_t': '2648.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '60.56', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.1', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853384407 ms)
2025-02-17 16:59:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853444402 ms)
2025-02-17 16:59:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853444402 ms)
2025-02-17 16:59:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 16:59:25.400177+00:00', 'runtime': '853440.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.79', 'adc_vl_f': '14.569', 'adc_ic_f': '3.921', 'adc_il_f': '0.292', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.2', 'ahc_t': '2648.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.89', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853444404 ms)
2025-02-17 16:59:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:58:25.400266+00:00', 'runtime': '853380.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.735', 'adc_vl_f': '14.563', 'adc_ic_f': '3.969', 'adc_il_f': '0.309', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.2', 'ahc_t': '2648.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.159', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853444404 ms)
2025-02-17 17:00:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853504413 ms)
2025-02-17 17:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853504414 ms)
2025-02-17 17:00:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:00:25.400175+00:00', 'runtime': '853500.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.704', 'adc_vl_f': '14.569', 'adc_ic_f': '4.073', 'adc_il_f': '0.302', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.3', 'ahc_t': '2648.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '59.322', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.3', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853504415 ms)
2025-02-17 17:00:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 16:59:25.400177+00:00', 'runtime': '853440.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.79', 'adc_vl_f': '14.569', 'adc_ic_f': '3.921', 'adc_il_f': '0.292', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.2', 'ahc_t': '2648.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.89', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.2', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853504415 ms)
2025-02-17 17:01:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853564410 ms)
2025-02-17 17:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853564410 ms)
2025-02-17 17:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:01:25.400180+00:00', 'runtime': '853560.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '4.042', 'adc_il_f': '0.29', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.4', 'ahc_t': '2648.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '58.869', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.4', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853564412 ms)
2025-02-17 17:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:00:25.400175+00:00', 'runtime': '853500.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.704', 'adc_vl_f': '14.569', 'adc_ic_f': '4.073', 'adc_il_f': '0.302', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.3', 'ahc_t': '2648.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '59.322', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.3', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853564412 ms)
2025-02-17 17:01:35.909 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853573729 ms)
2025-02-17 17:01:35.910 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853573729 ms)
2025-02-17 17:01:35.911 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:01:25.400344+00:00'} (T+853573730 ms)
2025-02-17 17:02:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853624406 ms)
2025-02-17 17:02:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853624407 ms)
2025-02-17 17:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:02:25.400173+00:00', 'runtime': '853620.001', 'adc_vb_f': '14.566', 'adc_va_f': '21.649', 'adc_vl_f': '14.529', 'adc_ic_f': '3.952', 'adc_il_f': '0.292', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2648.4', 'ahc_t': '2648.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '58.477', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.4', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853624408 ms)
2025-02-17 17:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:01:25.400180+00:00', 'runtime': '853560.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '4.042', 'adc_il_f': '0.29', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.4', 'ahc_t': '2648.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '58.869', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.4', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853624409 ms)
2025-02-17 17:03:12.901 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+853670720 ms)
2025-02-17 17:03:12.901 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+853670720 ms)
2025-02-17 17:03:12.902 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46730)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+853670721 ms)
2025-02-17 17:03:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853684402 ms)
2025-02-17 17:03:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853684403 ms)
2025-02-17 17:03:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:03:25.400942+00:00', 'runtime': '853680.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.548', 'adc_vl_f': '14.569', 'adc_ic_f': '3.981', 'adc_il_f': '0.295', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.5', 'ahc_t': '2648.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '58.024', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.5', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853684404 ms)
2025-02-17 17:03:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:02:25.400173+00:00', 'runtime': '853620.001', 'adc_vb_f': '14.566', 'adc_va_f': '21.649', 'adc_vl_f': '14.529', 'adc_ic_f': '3.952', 'adc_il_f': '0.292', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2648.4', 'ahc_t': '2648.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '58.477', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.4', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853684404 ms)
2025-02-17 17:04:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853744414 ms)
2025-02-17 17:04:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853744415 ms)
2025-02-17 17:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:04:25.400636+00:00', 'runtime': '853740.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.563', 'adc_ic_f': '3.955', 'adc_il_f': '0.302', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.6', 'ahc_t': '2648.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '57.631', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.6', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853744416 ms)
2025-02-17 17:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:03:25.400942+00:00', 'runtime': '853680.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.548', 'adc_vl_f': '14.569', 'adc_ic_f': '3.981', 'adc_il_f': '0.295', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.5', 'ahc_t': '2648.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '58.024', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.5', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853744416 ms)
2025-02-17 17:05:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853804409 ms)
2025-02-17 17:05:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853804410 ms)
2025-02-17 17:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:05:25.400193+00:00', 'runtime': '853800.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.619', 'adc_vl_f': '14.569', 'adc_ic_f': '3.94', 'adc_il_f': '0.326', 't_hs': '16', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.6', 'ahc_t': '2648.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '57.344', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.6', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853804411 ms)
2025-02-17 17:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:04:25.400636+00:00', 'runtime': '853740.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.558', 'adc_vl_f': '14.563', 'adc_ic_f': '3.955', 'adc_il_f': '0.302', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.6', 'ahc_t': '2648.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '57.631', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.6', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853804411 ms)
2025-02-17 17:05:35.813 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853813633 ms)
2025-02-17 17:05:35.814 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853813633 ms)
2025-02-17 17:05:35.816 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:05:25.405066+00:00'} (T+853813636 ms)
2025-02-17 17:06:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853864406 ms)
2025-02-17 17:06:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853864406 ms)
2025-02-17 17:06:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:06:25.400007+00:00', 'runtime': '853860.001', 'adc_vb_f': '14.572', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.877', 'adc_il_f': '0.314', 't_hs': '16', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2648.7', 'ahc_t': '2648.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '56.514', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.7', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853864407 ms)
2025-02-17 17:06:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:05:25.400193+00:00', 'runtime': '853800.001', 'adc_vb_f': '14.575', 'adc_va_f': '21.619', 'adc_vl_f': '14.569', 'adc_ic_f': '3.94', 'adc_il_f': '0.326', 't_hs': '16', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.6', 'ahc_t': '2648.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '57.344', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.6', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853864408 ms)
2025-02-17 17:06:35.878 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853873698 ms)
2025-02-17 17:06:35.879 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853873698 ms)
2025-02-17 17:06:35.880 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:06:25.405135+00:00'} (T+853873699 ms)
2025-02-17 17:07:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853924417 ms)
2025-02-17 17:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853924418 ms)
2025-02-17 17:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:07:25.402791+00:00', 'runtime': '853920.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '3.88', 'adc_il_f': '0.324', 't_hs': '16', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.8', 'ahc_t': '2648.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '56.499', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.8', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853924419 ms)
2025-02-17 17:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:06:25.400007+00:00', 'runtime': '853860.001', 'adc_vb_f': '14.572', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.877', 'adc_il_f': '0.314', 't_hs': '16', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.572', 'vb_ref': '14.089', 'ahc_r': '2648.7', 'ahc_t': '2648.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '56.514', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.7', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853924419 ms)
2025-02-17 17:07:35.621 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853933440 ms)
2025-02-17 17:07:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853933440 ms)
2025-02-17 17:07:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:07:25.400216+00:00'} (T+853933442 ms)
2025-02-17 17:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+853984415 ms)
2025-02-17 17:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+853984415 ms)
2025-02-17 17:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:08:25.400201+00:00', 'runtime': '853980.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.616', 'adc_vl_f': '14.526', 'adc_ic_f': '3.858', 'adc_il_f': '0.333', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.554', 'vb_ref': '14.06', 'ahc_r': '2648.8', 'ahc_t': '2648.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '56.076', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.8', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853984416 ms)
2025-02-17 17:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:07:25.402791+00:00', 'runtime': '853920.003', 'adc_vb_f': '14.575', 'adc_va_f': '21.664', 'adc_vl_f': '14.569', 'adc_ic_f': '3.88', 'adc_il_f': '0.324', 't_hs': '16', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.575', 'vb_ref': '14.089', 'ahc_r': '2648.8', 'ahc_t': '2648.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '56.499', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.8', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+853984416 ms)
2025-02-17 17:08:35.736 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853993555 ms)
2025-02-17 17:08:35.737 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+853993556 ms)
2025-02-17 17:08:35.738 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:08:25.400202+00:00'} (T+853993557 ms)
2025-02-17 17:09:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854044410 ms)
2025-02-17 17:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854044410 ms)
2025-02-17 17:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:09:25.402641+00:00', 'runtime': '854040.003', 'adc_vb_f': '14.532', 'adc_va_f': '21.732', 'adc_vl_f': '14.502', 'adc_ic_f': '3.706', 'adc_il_f': '0.326', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2648.9', 'ahc_t': '2648.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '55.563', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.9', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854044411 ms)
2025-02-17 17:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:08:25.400201+00:00', 'runtime': '853980.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.616', 'adc_vl_f': '14.526', 'adc_ic_f': '3.858', 'adc_il_f': '0.333', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.554', 'vb_ref': '14.06', 'ahc_r': '2648.8', 'ahc_t': '2648.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '56.076', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.8', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854044412 ms)
2025-02-17 17:09:35.768 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854053587 ms)
2025-02-17 17:09:35.768 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854053588 ms)
2025-02-17 17:09:35.770 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:09:25.400061+00:00'} (T+854053589 ms)
2025-02-17 17:09:55.459 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+854073279 ms)
2025-02-17 17:09:55.460 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+854073279 ms)
2025-02-17 17:09:55.460 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47168)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+854073279 ms)
2025-02-17 17:10:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854104407 ms)
2025-02-17 17:10:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854104408 ms)
2025-02-17 17:10:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:10:25.400014+00:00', 'runtime': '854100.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.529', 'adc_ic_f': '3.8', 'adc_il_f': '0.302', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.0', 'ahc_t': '2649.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '55.246', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854104409 ms)
2025-02-17 17:10:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:09:25.402641+00:00', 'runtime': '854040.003', 'adc_vb_f': '14.532', 'adc_va_f': '21.732', 'adc_vl_f': '14.502', 'adc_ic_f': '3.706', 'adc_il_f': '0.326', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2648.9', 'ahc_t': '2648.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.2', 'ahl_t': '1591.2', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '55.563', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '18.9', 'ahl_daily': '7.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854104409 ms)
2025-02-17 17:10:36.016 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854113835 ms)
2025-02-17 17:10:36.016 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854113836 ms)
2025-02-17 17:10:36.017 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:10:25.405127+00:00'} (T+854113837 ms)
2025-02-17 17:10:48.585 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+854126404 ms)
2025-02-17 17:10:48.585 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+854126404 ms)
2025-02-17 17:10:48.586 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47224)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+854126405 ms)
2025-02-17 17:11:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854164421 ms)
2025-02-17 17:11:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854164421 ms)
2025-02-17 17:11:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:11:25.400261+00:00', 'runtime': '854160.001', 'adc_vb_f': '14.532', 'adc_va_f': '21.863', 'adc_vl_f': '14.511', 'adc_ic_f': '3.624', 'adc_il_f': '0.302', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2649.0', 'ahc_t': '2649.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.566', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854164422 ms)
2025-02-17 17:11:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:10:25.400014+00:00', 'runtime': '854100.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.529', 'adc_ic_f': '3.8', 'adc_il_f': '0.302', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.0', 'ahc_t': '2649.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '55.246', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854164422 ms)
2025-02-17 17:11:36.324 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854174143 ms)
2025-02-17 17:11:36.325 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854174144 ms)
2025-02-17 17:11:36.326 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:11:25.413121+00:00'} (T+854174146 ms)
2025-02-17 17:12:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854224417 ms)
2025-02-17 17:12:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854224417 ms)
2025-02-17 17:12:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:12:25.400156+00:00', 'runtime': '854220.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.793', 'adc_vl_f': '14.529', 'adc_ic_f': '3.665', 'adc_il_f': '0.312', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.1', 'ahc_t': '2649.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.109', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854224418 ms)
2025-02-17 17:12:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:11:25.400261+00:00', 'runtime': '854160.001', 'adc_vb_f': '14.532', 'adc_va_f': '21.863', 'adc_vl_f': '14.511', 'adc_ic_f': '3.624', 'adc_il_f': '0.302', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2649.0', 'ahc_t': '2649.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.566', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854224419 ms)
2025-02-17 17:12:35.750 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854233570 ms)
2025-02-17 17:12:35.751 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854233570 ms)
2025-02-17 17:12:35.753 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:12:25.399999+00:00'} (T+854233572 ms)
2025-02-17 17:13:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854284413 ms)
2025-02-17 17:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854284414 ms)
2025-02-17 17:13:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:13:25.401135+00:00', 'runtime': '854280.002', 'adc_vb_f': '14.536', 'adc_va_f': '21.771', 'adc_vl_f': '14.514', 'adc_ic_f': '3.728', 'adc_il_f': '0.348', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.1', 'ahc_t': '2649.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.642', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854284415 ms)
2025-02-17 17:13:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:12:25.400156+00:00', 'runtime': '854220.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.793', 'adc_vl_f': '14.529', 'adc_ic_f': '3.665', 'adc_il_f': '0.312', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.1', 'ahc_t': '2649.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.109', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854284416 ms)
2025-02-17 17:13:35.809 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854293629 ms)
2025-02-17 17:13:35.810 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854293629 ms)
2025-02-17 17:13:35.811 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:13:25.400053+00:00'} (T+854293630 ms)
2025-02-17 17:14:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854344413 ms)
2025-02-17 17:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854344413 ms)
2025-02-17 17:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:14:25.400162+00:00', 'runtime': '854340.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.529', 'adc_ic_f': '3.718', 'adc_il_f': '0.3', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.2', 'ahc_t': '2649.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.083', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.2', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854344414 ms)
2025-02-17 17:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:13:25.401135+00:00', 'runtime': '854280.002', 'adc_vb_f': '14.536', 'adc_va_f': '21.771', 'adc_vl_f': '14.514', 'adc_ic_f': '3.728', 'adc_il_f': '0.348', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.1', 'ahc_t': '2649.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.642', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854344415 ms)
2025-02-17 17:14:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854353430 ms)
2025-02-17 17:14:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854353431 ms)
2025-02-17 17:14:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:14:25.400001+00:00'} (T+854353432 ms)
2025-02-17 17:15:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854404410 ms)
2025-02-17 17:15:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854404411 ms)
2025-02-17 17:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:15:25.400199+00:00', 'runtime': '854400.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.529', 'adc_ic_f': '3.679', 'adc_il_f': '0.312', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.3', 'ahc_t': '2649.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '53.615', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854404412 ms)
2025-02-17 17:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:14:25.400162+00:00', 'runtime': '854340.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.529', 'adc_ic_f': '3.718', 'adc_il_f': '0.3', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.2', 'ahc_t': '2649.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '54.083', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.2', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854404412 ms)
2025-02-17 17:15:35.738 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854413557 ms)
2025-02-17 17:15:35.738 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854413558 ms)
2025-02-17 17:15:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:15:25.400215+00:00'} (T+854413558 ms)
2025-02-17 17:16:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854464405 ms)
2025-02-17 17:16:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854464406 ms)
2025-02-17 17:16:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:16:25.400208+00:00', 'runtime': '854460.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.796', 'adc_vl_f': '14.529', 'adc_ic_f': '3.636', 'adc_il_f': '0.329', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.3', 'ahc_t': '2649.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.92', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854464407 ms)
2025-02-17 17:16:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:15:25.400199+00:00', 'runtime': '854400.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.529', 'adc_ic_f': '3.679', 'adc_il_f': '0.312', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.3', 'ahc_t': '2649.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '53.615', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854464407 ms)
2025-02-17 17:16:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854473429 ms)
2025-02-17 17:16:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854473429 ms)
2025-02-17 17:16:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:16:25.400070+00:00'} (T+854473431 ms)
2025-02-17 17:17:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854524417 ms)
2025-02-17 17:17:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854524417 ms)
2025-02-17 17:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:17:25.400209+00:00', 'runtime': '854520.001', 'adc_vb_f': '14.536', 'adc_va_f': '21.875', 'adc_vl_f': '14.52', 'adc_ic_f': '3.476', 'adc_il_f': '0.319', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2649.4', 'ahc_t': '2649.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.154', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.4', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854524418 ms)
2025-02-17 17:17:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:16:25.400208+00:00', 'runtime': '854460.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.796', 'adc_vl_f': '14.529', 'adc_ic_f': '3.636', 'adc_il_f': '0.329', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.3', 'ahc_t': '2649.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.92', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854524419 ms)
2025-02-17 17:17:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854533425 ms)
2025-02-17 17:17:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854533426 ms)
2025-02-17 17:17:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:17:25.400225+00:00'} (T+854533430 ms)
2025-02-17 17:18:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854584415 ms)
2025-02-17 17:18:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854584416 ms)
2025-02-17 17:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:18:25.400160+00:00', 'runtime': '854580.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.805', 'adc_vl_f': '14.529', 'adc_ic_f': '3.621', 'adc_il_f': '0.341', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.5', 'ahc_t': '2649.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.603', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854584417 ms)
2025-02-17 17:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:17:25.400209+00:00', 'runtime': '854520.001', 'adc_vb_f': '14.536', 'adc_va_f': '21.875', 'adc_vl_f': '14.52', 'adc_ic_f': '3.476', 'adc_il_f': '0.319', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2649.4', 'ahc_t': '2649.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.154', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.4', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854584418 ms)
2025-02-17 17:18:35.650 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854593469 ms)
2025-02-17 17:18:35.651 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854593470 ms)
2025-02-17 17:18:35.655 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:18:25.400174+00:00'} (T+854593475 ms)
2025-02-17 17:19:18.307 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+854636126 ms)
2025-02-17 17:19:18.307 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+854636126 ms)
2025-02-17 17:19:18.308 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47778)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+854636127 ms)
2025-02-17 17:19:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854644416 ms)
2025-02-17 17:19:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854644416 ms)
2025-02-17 17:19:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:19:25.400195+00:00', 'runtime': '854640.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.796', 'adc_vl_f': '14.529', 'adc_ic_f': '3.59', 'adc_il_f': '0.331', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.5', 'ahc_t': '2649.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.181', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854644417 ms)
2025-02-17 17:19:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:18:25.400160+00:00', 'runtime': '854580.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.805', 'adc_vl_f': '14.529', 'adc_ic_f': '3.621', 'adc_il_f': '0.341', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.5', 'ahc_t': '2649.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.603', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854644418 ms)
2025-02-17 17:19:35.844 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854653664 ms)
2025-02-17 17:19:35.845 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854653664 ms)
2025-02-17 17:19:35.846 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:19:25.400188+00:00'} (T+854653665 ms)
2025-02-17 17:20:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854704410 ms)
2025-02-17 17:20:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854704410 ms)
2025-02-17 17:20:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:20:25.400191+00:00', 'runtime': '854700.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.774', 'adc_vl_f': '14.523', 'adc_ic_f': '3.493', 'adc_il_f': '0.316', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.6', 'ahc_t': '2649.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.365', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.6', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854704411 ms)
2025-02-17 17:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:19:25.400195+00:00', 'runtime': '854640.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.796', 'adc_vl_f': '14.529', 'adc_ic_f': '3.59', 'adc_il_f': '0.331', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.5', 'ahc_t': '2649.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '52.181', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854704412 ms)
2025-02-17 17:20:35.657 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854713476 ms)
2025-02-17 17:20:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854713477 ms)
2025-02-17 17:20:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:20:25.400475+00:00'} (T+854713478 ms)
2025-02-17 17:20:42.485 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+854720305 ms)
2025-02-17 17:20:42.486 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+854720305 ms)
2025-02-17 17:20:42.486 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47870)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+854720305 ms)
2025-02-17 17:21:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854764416 ms)
2025-02-17 17:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854764417 ms)
2025-02-17 17:21:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:21:25.402734+00:00', 'runtime': '854760.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.725', 'adc_vl_f': '14.529', 'adc_ic_f': '3.534', 'adc_il_f': '0.287', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.6', 'ahc_t': '2649.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.38', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.6', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854764418 ms)
2025-02-17 17:21:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:20:25.400191+00:00', 'runtime': '854700.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.774', 'adc_vl_f': '14.523', 'adc_ic_f': '3.493', 'adc_il_f': '0.316', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.6', 'ahc_t': '2649.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.365', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.6', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854764421 ms)
2025-02-17 17:21:35.666 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854773486 ms)
2025-02-17 17:21:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854773486 ms)
2025-02-17 17:21:35.668 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:21:25.400150+00:00'} (T+854773488 ms)
2025-02-17 17:22:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854824420 ms)
2025-02-17 17:22:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854824421 ms)
2025-02-17 17:22:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:22:25.402802+00:00', 'runtime': '854820.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.474', 'adc_il_f': '0.283', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.7', 'ahc_t': '2649.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.459', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.7', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854824422 ms)
2025-02-17 17:22:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:21:25.402734+00:00', 'runtime': '854760.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.725', 'adc_vl_f': '14.529', 'adc_ic_f': '3.534', 'adc_il_f': '0.287', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.6', 'ahc_t': '2649.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '51.38', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.6', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854824422 ms)
2025-02-17 17:22:35.766 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854833585 ms)
2025-02-17 17:22:35.766 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854833586 ms)
2025-02-17 17:22:35.768 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:22:25.400214+00:00'} (T+854833587 ms)
2025-02-17 17:23:11.747 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+854869566 ms)
2025-02-17 17:23:11.747 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+854869566 ms)
2025-02-17 17:23:11.748 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48030)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+854869567 ms)
2025-02-17 17:23:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854884417 ms)
2025-02-17 17:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854884418 ms)
2025-02-17 17:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:23:25.402612+00:00', 'runtime': '854880.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.464', 'adc_il_f': '0.312', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.7', 'ahc_t': '2649.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.324', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.7', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854884419 ms)
2025-02-17 17:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:22:25.402802+00:00', 'runtime': '854820.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.474', 'adc_il_f': '0.283', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.7', 'ahc_t': '2649.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.459', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.7', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854884419 ms)
2025-02-17 17:23:35.898 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854893717 ms)
2025-02-17 17:23:35.898 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854893717 ms)
2025-02-17 17:23:35.900 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:23:25.400022+00:00'} (T+854893720 ms)
2025-02-17 17:24:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+854944417 ms)
2025-02-17 17:24:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+854944417 ms)
2025-02-17 17:24:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:24:25.402850+00:00', 'runtime': '854940.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.452', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.8', 'ahc_t': '2649.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.142', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.8', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854944418 ms)
2025-02-17 17:24:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:23:25.402612+00:00', 'runtime': '854880.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.464', 'adc_il_f': '0.312', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.7', 'ahc_t': '2649.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.324', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.7', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+854944419 ms)
2025-02-17 17:24:35.621 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854953441 ms)
2025-02-17 17:24:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+854953441 ms)
2025-02-17 17:24:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:24:25.400276+00:00'} (T+854953443 ms)
2025-02-17 17:25:07.972 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+854985792 ms)
2025-02-17 17:25:07.973 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+854985792 ms)
2025-02-17 17:25:07.973 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48156)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+854985792 ms)
2025-02-17 17:25:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855004414 ms)
2025-02-17 17:25:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855004415 ms)
2025-02-17 17:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:25:25.402838+00:00', 'runtime': '855000.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.523', 'adc_ic_f': '3.452', 'adc_il_f': '0.338', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.9', 'ahc_t': '2649.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.354', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.9', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855004416 ms)
2025-02-17 17:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:24:25.402850+00:00', 'runtime': '854940.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.529', 'adc_ic_f': '3.452', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.8', 'ahc_t': '2649.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.142', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.8', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855004417 ms)
2025-02-17 17:26:07.124 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+855044944 ms)
2025-02-17 17:26:07.125 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+855044944 ms)
2025-02-17 17:26:07.125 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48230)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855044944 ms)
2025-02-17 17:26:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855064415 ms)
2025-02-17 17:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855064415 ms)
2025-02-17 17:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:26:25.400656+00:00', 'runtime': '855060.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.765', 'adc_vl_f': '14.523', 'adc_ic_f': '3.382', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.9', 'ahc_t': '2649.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '49.463', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.9', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855064417 ms)
2025-02-17 17:26:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:25:25.402838+00:00', 'runtime': '855000.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.664', 'adc_vl_f': '14.523', 'adc_ic_f': '3.452', 'adc_il_f': '0.338', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.9', 'ahc_t': '2649.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.354', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.9', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855064417 ms)
2025-02-17 17:27:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855124413 ms)
2025-02-17 17:27:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855124414 ms)
2025-02-17 17:27:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:27:25.400593+00:00', 'runtime': '855120.001', 'adc_vb_f': '14.536', 'adc_va_f': '21.771', 'adc_vl_f': '14.523', 'adc_ic_f': '3.401', 'adc_il_f': '0.343', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2650.0', 'ahc_t': '2650.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.037', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855124415 ms)
2025-02-17 17:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:26:25.400656+00:00', 'runtime': '855060.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.765', 'adc_vl_f': '14.523', 'adc_ic_f': '3.382', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2649.9', 'ahc_t': '2649.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '49.463', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '19.9', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855124415 ms)
2025-02-17 17:28:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855184410 ms)
2025-02-17 17:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855184411 ms)
2025-02-17 17:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:28:25.402738+00:00', 'runtime': '855180.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.875', 'adc_vl_f': '14.529', 'adc_ic_f': '3.343', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.0', 'ahc_t': '2650.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.602', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855184412 ms)
2025-02-17 17:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:27:25.400593+00:00', 'runtime': '855120.001', 'adc_vb_f': '14.536', 'adc_va_f': '21.771', 'adc_vl_f': '14.523', 'adc_ic_f': '3.401', 'adc_il_f': '0.343', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2650.0', 'ahc_t': '2650.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.037', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855184413 ms)
2025-02-17 17:29:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855244409 ms)
2025-02-17 17:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855244409 ms)
2025-02-17 17:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:29:25.400193+00:00', 'runtime': '855240.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.878', 'adc_vl_f': '14.529', 'adc_ic_f': '3.353', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.1', 'ahc_t': '2650.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.738', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855244411 ms)
2025-02-17 17:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:28:25.402738+00:00', 'runtime': '855180.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.875', 'adc_vl_f': '14.529', 'adc_ic_f': '3.343', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.0', 'ahc_t': '2650.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.602', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.0', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855244411 ms)
2025-02-17 17:30:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855304405 ms)
2025-02-17 17:30:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855304406 ms)
2025-02-17 17:30:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:30:25.400183+00:00', 'runtime': '855300.001', 'adc_vb_f': '14.536', 'adc_va_f': '21.887', 'adc_vl_f': '14.52', 'adc_ic_f': '3.22', 'adc_il_f': '0.302', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2650.1', 'ahc_t': '2650.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.134', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855304408 ms)
2025-02-17 17:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:29:25.400193+00:00', 'runtime': '855240.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.878', 'adc_vl_f': '14.529', 'adc_ic_f': '3.353', 'adc_il_f': '0.321', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.1', 'ahc_t': '2650.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.738', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855304408 ms)
2025-02-17 17:30:35.447 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855313266 ms)
2025-02-17 17:30:35.447 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855313267 ms)
2025-02-17 17:30:35.452 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:30:25.400348+00:00'} (T+855313271 ms)
2025-02-17 17:31:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855364403 ms)
2025-02-17 17:31:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855364404 ms)
2025-02-17 17:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:31:25.400201+00:00', 'runtime': '855360.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.774', 'adc_vl_f': '14.529', 'adc_ic_f': '3.312', 'adc_il_f': '0.295', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.2', 'ahc_t': '2650.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.149', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.2', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855364408 ms)
2025-02-17 17:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:30:25.400183+00:00', 'runtime': '855300.001', 'adc_vb_f': '14.536', 'adc_va_f': '21.887', 'adc_vl_f': '14.52', 'adc_ic_f': '3.22', 'adc_il_f': '0.302', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2650.1', 'ahc_t': '2650.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.134', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.1', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855364408 ms)
2025-02-17 17:31:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855373436 ms)
2025-02-17 17:31:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855373437 ms)
2025-02-17 17:31:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:31:25.400374+00:00'} (T+855373438 ms)
2025-02-17 17:32:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855424417 ms)
2025-02-17 17:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855424417 ms)
2025-02-17 17:32:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:32:25.402822+00:00', 'runtime': '855420.003', 'adc_vb_f': '14.532', 'adc_va_f': '21.771', 'adc_vl_f': '14.511', 'adc_ic_f': '3.124', 'adc_il_f': '0.263', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2650.3', 'ahc_t': '2650.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '47.108', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855424418 ms)
2025-02-17 17:32:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:31:25.400201+00:00', 'runtime': '855360.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.774', 'adc_vl_f': '14.529', 'adc_ic_f': '3.312', 'adc_il_f': '0.295', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.2', 'ahc_t': '2650.2', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.149', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.2', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855424421 ms)
2025-02-17 17:32:31.655 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+855429474 ms)
2025-02-17 17:32:31.655 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+855429474 ms)
2025-02-17 17:32:31.655 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48634)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855429475 ms)
2025-02-17 17:32:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855433438 ms)
2025-02-17 17:32:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855433439 ms)
2025-02-17 17:32:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:32:25.400192+00:00'} (T+855433440 ms)
2025-02-17 17:33:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855484415 ms)
2025-02-17 17:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855484416 ms)
2025-02-17 17:33:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:33:25.402828+00:00', 'runtime': '855480.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.701', 'adc_vl_f': '14.529', 'adc_ic_f': '3.237', 'adc_il_f': '0.295', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.3', 'ahc_t': '2650.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '47.062', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855484420 ms)
2025-02-17 17:33:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:32:25.402822+00:00', 'runtime': '855420.003', 'adc_vb_f': '14.532', 'adc_va_f': '21.771', 'adc_vl_f': '14.511', 'adc_ic_f': '3.124', 'adc_il_f': '0.263', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.06', 'ahc_r': '2650.3', 'ahc_t': '2650.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '47.108', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855484420 ms)
2025-02-17 17:33:36.081 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855493901 ms)
2025-02-17 17:33:36.082 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855493901 ms)
2025-02-17 17:33:36.083 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:33:25.400187+00:00'} (T+855493902 ms)
2025-02-17 17:34:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855544412 ms)
2025-02-17 17:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855544413 ms)
2025-02-17 17:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:34:25.400150+00:00', 'runtime': '855540.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.765', 'adc_vl_f': '14.529', 'adc_ic_f': '3.252', 'adc_il_f': '0.307', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.4', 'ahc_t': '2650.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '47.274', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.4', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855544414 ms)
2025-02-17 17:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:33:25.402828+00:00', 'runtime': '855480.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.701', 'adc_vl_f': '14.529', 'adc_ic_f': '3.237', 'adc_il_f': '0.295', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.3', 'ahc_t': '2650.3', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '47.062', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.3', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855544414 ms)
2025-02-17 17:34:35.954 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855553774 ms)
2025-02-17 17:34:35.955 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855553774 ms)
2025-02-17 17:34:35.956 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:34:25.413119+00:00'} (T+855553775 ms)
2025-02-17 17:35:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855604412 ms)
2025-02-17 17:35:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855604412 ms)
2025-02-17 17:35:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:35:25.402840+00:00', 'runtime': '855600.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.526', 'adc_ic_f': '3.136', 'adc_il_f': '0.304', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.4', 'ahc_t': '2650.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.59', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.4', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855604416 ms)
2025-02-17 17:35:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:34:25.400150+00:00', 'runtime': '855540.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.765', 'adc_vl_f': '14.529', 'adc_ic_f': '3.252', 'adc_il_f': '0.307', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.4', 'ahc_t': '2650.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '47.274', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.4', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855604417 ms)
2025-02-17 17:36:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855664411 ms)
2025-02-17 17:36:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855664411 ms)
2025-02-17 17:36:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:36:25.400185+00:00', 'runtime': '855660.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.667', 'adc_vl_f': '14.529', 'adc_ic_f': '3.186', 'adc_il_f': '0.3', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.5', 'ahc_t': '2650.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.247', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855664412 ms)
2025-02-17 17:36:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:35:25.402840+00:00', 'runtime': '855600.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.771', 'adc_vl_f': '14.526', 'adc_ic_f': '3.136', 'adc_il_f': '0.304', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.4', 'ahc_t': '2650.4', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.59', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.4', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855664413 ms)
2025-02-17 17:36:51.074 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+855688894 ms)
2025-02-17 17:36:51.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+855688894 ms)
2025-02-17 17:36:51.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48920)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855688894 ms)
2025-02-17 17:37:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855724409 ms)
2025-02-17 17:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855724409 ms)
2025-02-17 17:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:37:25.400193+00:00', 'runtime': '855720.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.634', 'adc_vl_f': '14.529', 'adc_ic_f': '3.182', 'adc_il_f': '0.309', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.5', 'ahc_t': '2650.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.171', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855724411 ms)
2025-02-17 17:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:36:25.400185+00:00', 'runtime': '855660.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.667', 'adc_vl_f': '14.529', 'adc_ic_f': '3.186', 'adc_il_f': '0.3', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.5', 'ahc_t': '2650.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.247', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855724411 ms)
2025-02-17 17:37:29.123 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+855726942 ms)
2025-02-17 17:37:29.123 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+855726942 ms)
2025-02-17 17:37:29.124 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855726943 ms)
2025-02-17 17:37:29.124 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+855726944 ms)
2025-02-17 17:37:29.125 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+855726944 ms)
2025-02-17 17:37:29.125 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855726945 ms)
2025-02-17 17:37:29.127 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+855726946 ms)
2025-02-17 17:37:29.127 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+855726946 ms)
2025-02-17 17:37:29.127 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855726947 ms)
2025-02-17 17:37:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855733444 ms)
2025-02-17 17:37:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+855733445 ms)
2025-02-17 17:37:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:37:25.405067+00:00'} (T+855733446 ms)
2025-02-17 17:38:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855784420 ms)
2025-02-17 17:38:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855784421 ms)
2025-02-17 17:38:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:38:25.402798+00:00', 'runtime': '855780.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.658', 'adc_vl_f': '14.529', 'adc_ic_f': '3.143', 'adc_il_f': '0.314', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.6', 'ahc_t': '2650.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '45.688', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.6', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855784422 ms)
2025-02-17 17:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:37:25.400193+00:00', 'runtime': '855720.001', 'adc_vb_f': '14.539', 'adc_va_f': '21.634', 'adc_vl_f': '14.529', 'adc_ic_f': '3.182', 'adc_il_f': '0.309', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.5', 'ahc_t': '2650.5', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.171', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.5', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855784422 ms)
2025-02-17 17:38:48.294 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+855806113 ms)
2025-02-17 17:38:48.294 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+855806114 ms)
2025-02-17 17:38:48.295 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49046)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855806114 ms)
2025-02-17 17:39:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855844403 ms)
2025-02-17 17:39:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855844404 ms)
2025-02-17 17:39:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:39:25.400800+00:00', 'runtime': '855840.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.64', 'adc_vl_f': '14.493', 'adc_ic_f': '3.068', 'adc_il_f': '0.271', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.03', 'ahc_r': '2650.6', 'ahc_t': '2650.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.081', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855844406 ms)
2025-02-17 17:39:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:38:25.402798+00:00', 'runtime': '855780.003', 'adc_vb_f': '14.539', 'adc_va_f': '21.658', 'adc_vl_f': '14.529', 'adc_ic_f': '3.143', 'adc_il_f': '0.314', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.539', 'vb_ref': '14.06', 'ahc_r': '2650.6', 'ahc_t': '2650.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.3', 'ahl_t': '1591.3', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '45.688', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.6', 'ahl_daily': '7.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855844406 ms)
2025-02-17 17:39:29.423 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+855847242 ms)
2025-02-17 17:39:29.424 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+855847243 ms)
2025-02-17 17:39:29.424 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49090)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855847243 ms)
2025-02-17 17:39:32.504 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+855850323 ms)
2025-02-17 17:39:32.504 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+855850323 ms)
2025-02-17 17:39:32.504 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49094)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+855850324 ms)
2025-02-17 17:40:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855904401 ms)
2025-02-17 17:40:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855904401 ms)
2025-02-17 17:40:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:40:25.400594+00:00', 'runtime': '855900.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.558', 'adc_vl_f': '14.493', 'adc_ic_f': '3.049', 'adc_il_f': '0.251', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.514', 'vb_ref': '14.03', 'ahc_r': '2650.7', 'ahc_t': '2650.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.571', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855904402 ms)
2025-02-17 17:40:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:39:25.400800+00:00', 'runtime': '855840.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.64', 'adc_vl_f': '14.493', 'adc_ic_f': '3.068', 'adc_il_f': '0.271', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.536', 'vb_ref': '14.03', 'ahc_r': '2650.6', 'ahc_t': '2650.6', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.081', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855904403 ms)
2025-02-17 17:41:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+855964415 ms)
2025-02-17 17:41:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+855964415 ms)
2025-02-17 17:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:41:25.402772+00:00', 'runtime': '855960.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.683', 'adc_vl_f': '14.493', 'adc_ic_f': '3.054', 'adc_il_f': '0.258', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.7', 'ahc_t': '2650.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '45.266', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855964416 ms)
2025-02-17 17:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:40:25.400594+00:00', 'runtime': '855900.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.558', 'adc_vl_f': '14.493', 'adc_ic_f': '3.049', 'adc_il_f': '0.251', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.514', 'vb_ref': '14.03', 'ahc_r': '2650.7', 'ahc_t': '2650.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.571', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+855964417 ms)
2025-02-17 17:42:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856024412 ms)
2025-02-17 17:42:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856024413 ms)
2025-02-17 17:42:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:42:25.400716+00:00', 'runtime': '856020.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '3.075', 'adc_il_f': '0.3', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.8', 'ahc_t': '2650.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.148', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856024414 ms)
2025-02-17 17:42:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:41:25.402772+00:00', 'runtime': '855960.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.683', 'adc_vl_f': '14.493', 'adc_ic_f': '3.054', 'adc_il_f': '0.258', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.7', 'ahc_t': '2650.7', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '45.266', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856024415 ms)
2025-02-17 17:43:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856084409 ms)
2025-02-17 17:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856084409 ms)
2025-02-17 17:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:43:25.402781+00:00', 'runtime': '856080.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.869', 'adc_vl_f': '14.493', 'adc_ic_f': '3.032', 'adc_il_f': '0.292', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2650.8', 'ahc_t': '2650.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.571', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856084410 ms)
2025-02-17 17:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:42:25.400716+00:00', 'runtime': '856020.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '3.075', 'adc_il_f': '0.3', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.8', 'ahc_t': '2650.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.148', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856084411 ms)
2025-02-17 17:44:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856144409 ms)
2025-02-17 17:44:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856144409 ms)
2025-02-17 17:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:44:25.400581+00:00', 'runtime': '856140.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.823', 'adc_vl_f': '14.493', 'adc_ic_f': '3.017', 'adc_il_f': '0.285', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.9', 'ahc_t': '2650.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.484', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856144410 ms)
2025-02-17 17:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:43:25.402781+00:00', 'runtime': '856080.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.869', 'adc_vl_f': '14.493', 'adc_ic_f': '3.032', 'adc_il_f': '0.292', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2650.8', 'ahc_t': '2650.8', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.571', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856144411 ms)
2025-02-17 17:44:51.973 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+856169792 ms)
2025-02-17 17:44:51.973 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+856169792 ms)
2025-02-17 17:44:51.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49428)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+856169793 ms)
2025-02-17 17:45:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856204420 ms)
2025-02-17 17:45:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856204421 ms)
2025-02-17 17:45:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:45:25.403142+00:00', 'runtime': '856200.004', 'adc_vb_f': '14.508', 'adc_va_f': '21.777', 'adc_vl_f': '14.493', 'adc_ic_f': '3.022', 'adc_il_f': '0.324', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.9', 'ahc_t': '2650.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.329', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856204423 ms)
2025-02-17 17:45:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:44:25.400581+00:00', 'runtime': '856140.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.823', 'adc_vl_f': '14.493', 'adc_ic_f': '3.017', 'adc_il_f': '0.285', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.9', 'ahc_t': '2650.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.484', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856204423 ms)
2025-02-17 17:45:35.692 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856213512 ms)
2025-02-17 17:45:35.693 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856213512 ms)
2025-02-17 17:45:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:45:25.400465+00:00'} (T+856213513 ms)
2025-02-17 17:45:50.133 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+856227952 ms)
2025-02-17 17:45:50.133 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+856227953 ms)
2025-02-17 17:45:50.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49502)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+856227953 ms)
2025-02-17 17:46:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856264417 ms)
2025-02-17 17:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856264418 ms)
2025-02-17 17:46:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:46:25.400174+00:00', 'runtime': '856260.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.817', 'adc_vl_f': '14.493', 'adc_ic_f': '3.027', 'adc_il_f': '0.35', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.0', 'ahc_t': '2651.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.997', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856264419 ms)
2025-02-17 17:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:45:25.403142+00:00', 'runtime': '856200.004', 'adc_vb_f': '14.508', 'adc_va_f': '21.777', 'adc_vl_f': '14.493', 'adc_ic_f': '3.022', 'adc_il_f': '0.324', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2650.9', 'ahc_t': '2650.9', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.329', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '20.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856264419 ms)
2025-02-17 17:46:35.745 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856273564 ms)
2025-02-17 17:46:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856273565 ms)
2025-02-17 17:46:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:46:25.405140+00:00'} (T+856273566 ms)
2025-02-17 17:47:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856324415 ms)
2025-02-17 17:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856324416 ms)
2025-02-17 17:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:47:25.402684+00:00', 'runtime': '856320.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.993', 'adc_il_f': '0.307', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.0', 'ahc_t': '2651.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.288', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856324417 ms)
2025-02-17 17:47:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:46:25.400174+00:00', 'runtime': '856260.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.817', 'adc_vl_f': '14.493', 'adc_ic_f': '3.027', 'adc_il_f': '0.35', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.0', 'ahc_t': '2651.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.997', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856324417 ms)
2025-02-17 17:47:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856333434 ms)
2025-02-17 17:47:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856333434 ms)
2025-02-17 17:47:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:47:25.400049+00:00'} (T+856333435 ms)
2025-02-17 17:48:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856384410 ms)
2025-02-17 17:48:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856384411 ms)
2025-02-17 17:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:48:25.402783+00:00', 'runtime': '856380.003', 'adc_vb_f': '14.508', 'adc_va_f': '21.854', 'adc_vl_f': '14.493', 'adc_ic_f': '2.933', 'adc_il_f': '0.307', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.1', 'ahc_t': '2651.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.725', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856384412 ms)
2025-02-17 17:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:47:25.402684+00:00', 'runtime': '856320.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.993', 'adc_il_f': '0.307', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.0', 'ahc_t': '2651.0', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.288', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856384412 ms)
2025-02-17 17:48:54.433 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+856412253 ms)
2025-02-17 17:48:54.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+856412253 ms)
2025-02-17 17:48:54.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49700)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+856412254 ms)
2025-02-17 17:49:02.513 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+856420333 ms)
2025-02-17 17:49:02.514 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+856420333 ms)
2025-02-17 17:49:02.514 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49710)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+856420334 ms)
2025-02-17 17:49:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856444407 ms)
2025-02-17 17:49:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856444408 ms)
2025-02-17 17:49:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:49:25.400159+00:00', 'runtime': '856440.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.783', 'adc_vl_f': '14.493', 'adc_ic_f': '2.88', 'adc_il_f': '0.249', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.1', 'ahc_t': '2651.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.974', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856444409 ms)
2025-02-17 17:49:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:48:25.402783+00:00', 'runtime': '856380.003', 'adc_vb_f': '14.508', 'adc_va_f': '21.854', 'adc_vl_f': '14.493', 'adc_ic_f': '2.933', 'adc_il_f': '0.307', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.1', 'ahc_t': '2651.1', 'kwhc': '34.3', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.725', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856444409 ms)
2025-02-17 17:50:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856504405 ms)
2025-02-17 17:50:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856504406 ms)
2025-02-17 17:50:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:50:25.400169+00:00', 'runtime': '856500.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.786', 'adc_vl_f': '14.493', 'adc_ic_f': '2.894', 'adc_il_f': '0.266', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.2', 'ahc_t': '2651.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.91', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856504407 ms)
2025-02-17 17:50:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:49:25.400159+00:00', 'runtime': '856440.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.783', 'adc_vl_f': '14.493', 'adc_ic_f': '2.88', 'adc_il_f': '0.249', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.1', 'ahc_t': '2651.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.974', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856504407 ms)
2025-02-17 17:51:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856564403 ms)
2025-02-17 17:51:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856564404 ms)
2025-02-17 17:51:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:51:25.400179+00:00', 'runtime': '856560.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.843', 'adc_il_f': '0.242', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.2', 'ahc_t': '2651.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.627', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856564405 ms)
2025-02-17 17:51:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:50:25.400169+00:00', 'runtime': '856500.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.786', 'adc_vl_f': '14.493', 'adc_ic_f': '2.894', 'adc_il_f': '0.266', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.2', 'ahc_t': '2651.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.91', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856564405 ms)
2025-02-17 17:51:42.788 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+856580607 ms)
2025-02-17 17:51:42.788 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+856580607 ms)
2025-02-17 17:51:42.789 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49878)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+856580608 ms)
2025-02-17 17:52:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856624418 ms)
2025-02-17 17:52:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856624418 ms)
2025-02-17 17:52:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:52:25.400192+00:00', 'runtime': '856620.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.872', 'adc_il_f': '0.278', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.3', 'ahc_t': '2651.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.306', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856624422 ms)
2025-02-17 17:52:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:51:25.400179+00:00', 'runtime': '856560.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.843', 'adc_il_f': '0.242', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.2', 'ahc_t': '2651.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.627', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856624423 ms)
2025-02-17 17:53:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856684415 ms)
2025-02-17 17:53:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856684416 ms)
2025-02-17 17:53:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:53:25.400184+00:00', 'runtime': '856680.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.78', 'adc_vl_f': '14.493', 'adc_ic_f': '2.86', 'adc_il_f': '0.28', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.3', 'ahc_t': '2651.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.932', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856684420 ms)
2025-02-17 17:53:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:52:25.400192+00:00', 'runtime': '856620.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.872', 'adc_il_f': '0.278', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.3', 'ahc_t': '2651.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.306', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856684420 ms)
2025-02-17 17:53:39.002 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+856696821 ms)
2025-02-17 17:53:39.002 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+856696821 ms)
2025-02-17 17:53:39.003 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50002)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+856696822 ms)
2025-02-17 17:54:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856744414 ms)
2025-02-17 17:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856744415 ms)
2025-02-17 17:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:54:25.400171+00:00', 'runtime': '856740.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.765', 'adc_vl_f': '14.493', 'adc_ic_f': '2.884', 'adc_il_f': '0.345', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.4', 'ahc_t': '2651.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.125', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856744416 ms)
2025-02-17 17:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:53:25.400184+00:00', 'runtime': '856680.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.78', 'adc_vl_f': '14.493', 'adc_ic_f': '2.86', 'adc_il_f': '0.28', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.3', 'ahc_t': '2651.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20124', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.932', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856744417 ms)
2025-02-17 17:54:35.628 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856753447 ms)
2025-02-17 17:54:35.629 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856753448 ms)
2025-02-17 17:54:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:54:25.413050+00:00'} (T+856753449 ms)
2025-02-17 17:55:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856804411 ms)
2025-02-17 17:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856804411 ms)
2025-02-17 17:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:55:25.402745+00:00', 'runtime': '856800.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.863', 'adc_il_f': '0.309', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.4', 'ahc_t': '2651.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.521', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856804415 ms)
2025-02-17 17:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:54:25.400171+00:00', 'runtime': '856740.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.765', 'adc_vl_f': '14.493', 'adc_ic_f': '2.884', 'adc_il_f': '0.345', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.4', 'ahc_t': '2651.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.125', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856804416 ms)
2025-02-17 17:56:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856864410 ms)
2025-02-17 17:56:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856864411 ms)
2025-02-17 17:56:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:56:25.400160+00:00', 'runtime': '856860.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.834', 'adc_il_f': '0.295', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.5', 'ahc_t': '2651.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.736', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856864412 ms)
2025-02-17 17:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:55:25.402745+00:00', 'runtime': '856800.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.863', 'adc_il_f': '0.309', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.4', 'ahc_t': '2651.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.521', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856864414 ms)
2025-02-17 17:57:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856924409 ms)
2025-02-17 17:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856924410 ms)
2025-02-17 17:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:57:25.400194+00:00', 'runtime': '856920.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.777', 'adc_vl_f': '14.493', 'adc_ic_f': '2.822', 'adc_il_f': '0.297', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.5', 'ahc_t': '2651.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.144', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856924411 ms)
2025-02-17 17:57:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:56:25.400160+00:00', 'runtime': '856860.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.834', 'adc_il_f': '0.295', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.5', 'ahc_t': '2651.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.736', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856924413 ms)
2025-02-17 17:57:35.773 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856933592 ms)
2025-02-17 17:57:35.773 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856933592 ms)
2025-02-17 17:57:35.774 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:57:25.400349+00:00'} (T+856933593 ms)
2025-02-17 17:58:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+856984406 ms)
2025-02-17 17:58:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+856984407 ms)
2025-02-17 17:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:58:25.400209+00:00', 'runtime': '856980.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.79', 'adc_il_f': '0.29', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.6', 'ahc_t': '2651.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.751', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856984408 ms)
2025-02-17 17:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:57:25.400194+00:00', 'runtime': '856920.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.777', 'adc_vl_f': '14.493', 'adc_ic_f': '2.822', 'adc_il_f': '0.297', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.5', 'ahc_t': '2651.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.144', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+856984408 ms)
2025-02-17 17:58:29.412 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+856987231 ms)
2025-02-17 17:58:29.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+856987231 ms)
2025-02-17 17:58:29.413 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50320)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+856987232 ms)
2025-02-17 17:58:35.691 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856993511 ms)
2025-02-17 17:58:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+856993511 ms)
2025-02-17 17:58:35.693 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:58:25.400067+00:00'} (T+856993513 ms)
2025-02-17 17:58:43.502 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857001322 ms)
2025-02-17 17:58:43.503 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857001322 ms)
2025-02-17 17:58:43.503 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50338)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857001322 ms)
2025-02-17 17:59:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857044407 ms)
2025-02-17 17:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857044407 ms)
2025-02-17 17:59:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 17:59:25.400189+00:00', 'runtime': '857040.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.749', 'adc_il_f': '0.258', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.6', 'ahc_t': '2651.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.917', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857044408 ms)
2025-02-17 17:59:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:58:25.400209+00:00', 'runtime': '856980.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.79', 'adc_il_f': '0.29', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.6', 'ahc_t': '2651.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.751', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857044408 ms)
2025-02-17 17:59:29.631 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857047450 ms)
2025-02-17 17:59:29.631 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857047451 ms)
2025-02-17 17:59:29.632 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50388)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857047451 ms)
2025-02-17 17:59:35.856 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857053675 ms)
2025-02-17 17:59:35.856 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857053675 ms)
2025-02-17 17:59:35.857 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 17:59:25.400101+00:00'} (T+857053677 ms)
2025-02-17 18:00:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857104420 ms)
2025-02-17 18:00:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857104421 ms)
2025-02-17 18:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:00:25.402808+00:00', 'runtime': '857100.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.732', 'adc_il_f': '0.261', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.7', 'ahc_t': '2651.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.434', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857104422 ms)
2025-02-17 18:00:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 17:59:25.400189+00:00', 'runtime': '857040.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.749', 'adc_il_f': '0.258', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.6', 'ahc_t': '2651.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.917', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857104422 ms)
2025-02-17 18:00:48.772 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857126592 ms)
2025-02-17 18:00:48.773 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857126592 ms)
2025-02-17 18:00:48.773 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50474)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857126592 ms)
2025-02-17 18:01:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857164403 ms)
2025-02-17 18:01:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857164404 ms)
2025-02-17 18:01:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:01:25.400578+00:00', 'runtime': '857160.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.729', 'adc_vl_f': '14.493', 'adc_ic_f': '2.725', 'adc_il_f': '0.258', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.7', 'ahc_t': '2651.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.875', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857164405 ms)
2025-02-17 18:01:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:00:25.402808+00:00', 'runtime': '857100.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.732', 'adc_il_f': '0.261', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.7', 'ahc_t': '2651.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.434', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857164405 ms)
2025-02-17 18:02:26.581 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857224401 ms)
2025-02-17 18:02:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857224401 ms)
2025-02-17 18:02:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:02:25.400608+00:00', 'runtime': '857220.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.677', 'adc_vl_f': '14.493', 'adc_ic_f': '2.723', 'adc_il_f': '0.273', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.7', 'ahc_t': '2651.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.241', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857224402 ms)
2025-02-17 18:02:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:01:25.400578+00:00', 'runtime': '857160.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.729', 'adc_vl_f': '14.493', 'adc_ic_f': '2.725', 'adc_il_f': '0.258', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.7', 'ahc_t': '2651.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.875', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857224403 ms)
2025-02-17 18:03:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857284416 ms)
2025-02-17 18:03:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857284417 ms)
2025-02-17 18:03:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:03:25.400266+00:00', 'runtime': '857280.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.73', 'adc_il_f': '0.297', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.8', 'ahc_t': '2651.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.936', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857284419 ms)
2025-02-17 18:03:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:02:25.400608+00:00', 'runtime': '857220.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.677', 'adc_vl_f': '14.493', 'adc_ic_f': '2.723', 'adc_il_f': '0.273', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.7', 'ahc_t': '2651.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.241', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857284419 ms)
2025-02-17 18:03:29.041 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857286860 ms)
2025-02-17 18:03:29.041 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857286861 ms)
2025-02-17 18:03:29.043 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50642)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857286862 ms)
2025-02-17 18:03:35.657 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857293476 ms)
2025-02-17 18:03:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857293477 ms)
2025-02-17 18:03:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:03:25.413049+00:00'} (T+857293478 ms)
2025-02-17 18:03:52.139 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857309958 ms)
2025-02-17 18:03:52.139 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857309958 ms)
2025-02-17 18:03:52.140 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50670)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857309959 ms)
2025-02-17 18:04:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857344416 ms)
2025-02-17 18:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857344416 ms)
2025-02-17 18:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:04:25.402737+00:00', 'runtime': '857340.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.759', 'adc_vl_f': '14.493', 'adc_ic_f': '2.727', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.8', 'ahc_t': '2651.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.664', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857344417 ms)
2025-02-17 18:04:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:03:25.400266+00:00', 'runtime': '857280.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.73', 'adc_il_f': '0.297', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.8', 'ahc_t': '2651.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.936', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857344418 ms)
2025-02-17 18:04:35.706 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857353525 ms)
2025-02-17 18:04:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857353526 ms)
2025-02-17 18:04:35.707 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:04:25.400175+00:00'} (T+857353527 ms)
2025-02-17 18:05:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857404416 ms)
2025-02-17 18:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857404416 ms)
2025-02-17 18:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:05:25.402669+00:00', 'runtime': '857400.003', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.708', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.9', 'ahc_t': '2651.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.498', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857404417 ms)
2025-02-17 18:05:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:04:25.402737+00:00', 'runtime': '857340.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.759', 'adc_vl_f': '14.493', 'adc_ic_f': '2.727', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.8', 'ahc_t': '2651.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.664', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857404418 ms)
2025-02-17 18:06:11.395 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857449214 ms)
2025-02-17 18:06:11.395 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857449214 ms)
2025-02-17 18:06:11.395 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50828)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857449215 ms)
2025-02-17 18:06:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857464415 ms)
2025-02-17 18:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857464416 ms)
2025-02-17 18:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:06:25.400158+00:00', 'runtime': '857460.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.78', 'adc_vl_f': '14.493', 'adc_ic_f': '2.621', 'adc_il_f': '0.246', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.9', 'ahc_t': '2651.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.135', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857464417 ms)
2025-02-17 18:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:05:25.402669+00:00', 'runtime': '857400.003', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.708', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2651.9', 'ahc_t': '2651.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.498', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857464417 ms)
2025-02-17 18:06:30.486 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857468305 ms)
2025-02-17 18:06:30.486 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857468305 ms)
2025-02-17 18:06:30.487 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50848)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857468306 ms)
2025-02-17 18:06:59.616 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857497436 ms)
2025-02-17 18:06:59.617 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857497436 ms)
2025-02-17 18:06:59.617 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50880)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857497436 ms)
2025-02-17 18:07:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857524414 ms)
2025-02-17 18:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857524415 ms)
2025-02-17 18:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:07:25.400170+00:00', 'runtime': '857520.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.624', 'adc_il_f': '0.246', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.0', 'ahc_t': '2652.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.849', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857524419 ms)
2025-02-17 18:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:06:25.400158+00:00', 'runtime': '857460.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.78', 'adc_vl_f': '14.493', 'adc_ic_f': '2.621', 'adc_il_f': '0.246', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2651.9', 'ahc_t': '2651.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.135', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '21.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857524419 ms)
2025-02-17 18:08:02.779 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857560598 ms)
2025-02-17 18:08:02.779 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857560598 ms)
2025-02-17 18:08:02.779 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50948)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857560598 ms)
2025-02-17 18:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857584414 ms)
2025-02-17 18:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857584415 ms)
2025-02-17 18:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:08:25.400185+00:00', 'runtime': '857580.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.64', 'adc_il_f': '0.261', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.0', 'ahc_t': '2652.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.913', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857584416 ms)
2025-02-17 18:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:07:25.400170+00:00', 'runtime': '857520.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.624', 'adc_il_f': '0.246', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.0', 'ahc_t': '2652.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.849', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857584417 ms)
2025-02-17 18:09:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857644412 ms)
2025-02-17 18:09:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857644413 ms)
2025-02-17 18:09:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:09:25.400176+00:00', 'runtime': '857640.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.631', 'adc_il_f': '0.278', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.1', 'ahc_t': '2652.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.441', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857644415 ms)
2025-02-17 18:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:08:25.400185+00:00', 'runtime': '857580.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.64', 'adc_il_f': '0.261', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.0', 'ahc_t': '2652.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.913', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857644415 ms)
2025-02-17 18:09:35.659 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857653478 ms)
2025-02-17 18:09:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857653478 ms)
2025-02-17 18:09:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:09:25.400341+00:00'} (T+857653480 ms)
2025-02-17 18:10:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857704414 ms)
2025-02-17 18:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857704414 ms)
2025-02-17 18:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:10:25.400192+00:00', 'runtime': '857700.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.967', 'adc_vl_f': '14.493', 'adc_ic_f': '2.636', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.1', 'ahc_t': '2652.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.396', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857704419 ms)
2025-02-17 18:10:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:09:25.400176+00:00', 'runtime': '857640.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.631', 'adc_il_f': '0.278', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.1', 'ahc_t': '2652.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.441', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857704419 ms)
2025-02-17 18:10:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857713431 ms)
2025-02-17 18:10:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857713432 ms)
2025-02-17 18:10:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:10:25.400140+00:00'} (T+857713433 ms)
2025-02-17 18:11:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857764411 ms)
2025-02-17 18:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857764412 ms)
2025-02-17 18:11:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:11:25.400534+00:00', 'runtime': '857760.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.631', 'adc_il_f': '0.307', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.1', 'ahc_t': '2652.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.305', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857764416 ms)
2025-02-17 18:11:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:10:25.400192+00:00', 'runtime': '857700.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.967', 'adc_vl_f': '14.493', 'adc_ic_f': '2.636', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.1', 'ahc_t': '2652.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.396', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857764416 ms)
2025-02-17 18:12:15.183 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857813002 ms)
2025-02-17 18:12:15.183 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857813002 ms)
2025-02-17 18:12:15.183 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51216)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857813003 ms)
2025-02-17 18:12:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857824409 ms)
2025-02-17 18:12:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857824410 ms)
2025-02-17 18:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:12:25.400187+00:00', 'runtime': '857820.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.595', 'adc_il_f': '0.29', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.2', 'ahc_t': '2652.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.55', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857824413 ms)
2025-02-17 18:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:11:25.400534+00:00', 'runtime': '857760.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.631', 'adc_il_f': '0.307', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.1', 'ahc_t': '2652.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.305', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857824414 ms)
2025-02-17 18:13:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857884410 ms)
2025-02-17 18:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857884410 ms)
2025-02-17 18:13:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:13:25.400192+00:00', 'runtime': '857880.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.582', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.2', 'ahc_t': '2652.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.746', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857884411 ms)
2025-02-17 18:13:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:12:25.400187+00:00', 'runtime': '857820.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.595', 'adc_il_f': '0.29', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.2', 'ahc_t': '2652.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.55', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857884412 ms)
2025-02-17 18:13:29.352 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857887171 ms)
2025-02-17 18:13:29.352 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857887171 ms)
2025-02-17 18:13:29.352 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51294)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857887172 ms)
2025-02-17 18:13:35.830 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857893649 ms)
2025-02-17 18:13:35.831 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+857893650 ms)
2025-02-17 18:13:35.831 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:13:25.400347+00:00'} (T+857893651 ms)
2025-02-17 18:14:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+857944411 ms)
2025-02-17 18:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+857944412 ms)
2025-02-17 18:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:14:25.400181+00:00', 'runtime': '857940.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.896', 'adc_vl_f': '14.493', 'adc_ic_f': '2.546', 'adc_il_f': '0.266', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.3', 'ahc_t': '2652.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.656', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857944413 ms)
2025-02-17 18:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:13:25.400192+00:00', 'runtime': '857880.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.582', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.2', 'ahc_t': '2652.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.746', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+857944413 ms)
2025-02-17 18:14:58.564 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+857976383 ms)
2025-02-17 18:14:58.564 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+857976383 ms)
2025-02-17 18:14:58.565 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51392)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+857976384 ms)
2025-02-17 18:15:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858004409 ms)
2025-02-17 18:15:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858004409 ms)
2025-02-17 18:15:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:15:25.400202+00:00', 'runtime': '858000.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.793', 'adc_vl_f': '14.493', 'adc_ic_f': '2.532', 'adc_il_f': '0.234', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.3', 'ahc_t': '2652.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.659', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858004410 ms)
2025-02-17 18:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:14:25.400181+00:00', 'runtime': '857940.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.896', 'adc_vl_f': '14.493', 'adc_ic_f': '2.546', 'adc_il_f': '0.266', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.3', 'ahc_t': '2652.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.656', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858004411 ms)
2025-02-17 18:15:41.712 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+858019531 ms)
2025-02-17 18:15:41.712 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+858019531 ms)
2025-02-17 18:15:41.713 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51436)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858019532 ms)
2025-02-17 18:15:48.769 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+858026588 ms)
2025-02-17 18:15:48.769 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+858026589 ms)
2025-02-17 18:15:48.770 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51452)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858026590 ms)
2025-02-17 18:16:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858064406 ms)
2025-02-17 18:16:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858064406 ms)
2025-02-17 18:16:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:16:25.400259+00:00', 'runtime': '858060.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.722', 'adc_vl_f': '14.493', 'adc_ic_f': '2.57', 'adc_il_f': '0.297', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.4', 'ahc_t': '2652.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.49', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858064407 ms)
2025-02-17 18:16:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:15:25.400202+00:00', 'runtime': '858000.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.793', 'adc_vl_f': '14.493', 'adc_ic_f': '2.532', 'adc_il_f': '0.234', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.3', 'ahc_t': '2652.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.659', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858064408 ms)
2025-02-17 18:17:15.965 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+858113784 ms)
2025-02-17 18:17:15.965 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+858113784 ms)
2025-02-17 18:17:15.966 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51548)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858113785 ms)
2025-02-17 18:17:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858124404 ms)
2025-02-17 18:17:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858124404 ms)
2025-02-17 18:17:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:17:25.400168+00:00', 'runtime': '858120.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.556', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.4', 'ahc_t': '2652.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.825', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858124405 ms)
2025-02-17 18:17:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:16:25.400259+00:00', 'runtime': '858060.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.722', 'adc_vl_f': '14.493', 'adc_ic_f': '2.57', 'adc_il_f': '0.297', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.4', 'ahc_t': '2652.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.49', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858124406 ms)
2025-02-17 18:18:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858184402 ms)
2025-02-17 18:18:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858184402 ms)
2025-02-17 18:18:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:18:25.400178+00:00', 'runtime': '858180.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.522', 'adc_il_f': '0.268', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.4', 'ahc_t': '2652.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.403', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858184403 ms)
2025-02-17 18:18:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:17:25.400168+00:00', 'runtime': '858120.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.556', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.4', 'ahc_t': '2652.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.825', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858184404 ms)
2025-02-17 18:18:54.163 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+858211982 ms)
2025-02-17 18:18:54.163 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+858211982 ms)
2025-02-17 18:18:54.163 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51652)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858211983 ms)
2025-02-17 18:19:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858244403 ms)
2025-02-17 18:19:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858244404 ms)
2025-02-17 18:19:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:19:25.400227+00:00', 'runtime': '858240.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.606', 'adc_vl_f': '14.493', 'adc_ic_f': '2.488', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.5', 'ahc_t': '2652.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.931', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858244405 ms)
2025-02-17 18:19:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:18:25.400178+00:00', 'runtime': '858180.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.522', 'adc_il_f': '0.268', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.4', 'ahc_t': '2652.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.403', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.4', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858244406 ms)
2025-02-17 18:19:35.754 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858253573 ms)
2025-02-17 18:19:35.755 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858253574 ms)
2025-02-17 18:19:35.755 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:19:25.400349+00:00'} (T+858253575 ms)
2025-02-17 18:20:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858304418 ms)
2025-02-17 18:20:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858304418 ms)
2025-02-17 18:20:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:20:25.400173+00:00', 'runtime': '858300.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.649', 'adc_vl_f': '14.493', 'adc_ic_f': '2.491', 'adc_il_f': '0.278', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.5', 'ahc_t': '2652.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.342', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858304419 ms)
2025-02-17 18:20:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:19:25.400227+00:00', 'runtime': '858240.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.606', 'adc_vl_f': '14.493', 'adc_ic_f': '2.488', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.5', 'ahc_t': '2652.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.931', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858304420 ms)
2025-02-17 18:20:35.770 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858313589 ms)
2025-02-17 18:20:35.771 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858313590 ms)
2025-02-17 18:20:35.771 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:20:25.400061+00:00'} (T+858313591 ms)
2025-02-17 18:21:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858364416 ms)
2025-02-17 18:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858364416 ms)
2025-02-17 18:21:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:21:25.400162+00:00', 'runtime': '858360.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.625', 'adc_vl_f': '14.493', 'adc_ic_f': '2.464', 'adc_il_f': '0.263', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.6', 'ahc_t': '2652.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.98', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858364417 ms)
2025-02-17 18:21:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:20:25.400173+00:00', 'runtime': '858300.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.649', 'adc_vl_f': '14.493', 'adc_ic_f': '2.491', 'adc_il_f': '0.278', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.5', 'ahc_t': '2652.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.342', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.5', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858364418 ms)
2025-02-17 18:21:35.804 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858373623 ms)
2025-02-17 18:21:35.805 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858373624 ms)
2025-02-17 18:21:35.806 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:21:25.400028+00:00'} (T+858373625 ms)
2025-02-17 18:22:06.405 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+858404225 ms)
2025-02-17 18:22:06.406 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+858404225 ms)
2025-02-17 18:22:06.406 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858404226 ms)
2025-02-17 18:22:06.408 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+858404227 ms)
2025-02-17 18:22:06.408 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+858404227 ms)
2025-02-17 18:22:06.408 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858404228 ms)
2025-02-17 18:22:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858424415 ms)
2025-02-17 18:22:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858424416 ms)
2025-02-17 18:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:22:25.400159+00:00', 'runtime': '858420.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.643', 'adc_vl_f': '14.493', 'adc_ic_f': '2.454', 'adc_il_f': '0.249', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.6', 'ahc_t': '2652.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.161', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858424417 ms)
2025-02-17 18:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:21:25.400162+00:00', 'runtime': '858360.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.625', 'adc_vl_f': '14.493', 'adc_ic_f': '2.464', 'adc_il_f': '0.263', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.6', 'ahc_t': '2652.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.98', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858424417 ms)
2025-02-17 18:22:35.757 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858433576 ms)
2025-02-17 18:22:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858433577 ms)
2025-02-17 18:22:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:22:25.399991+00:00'} (T+858433578 ms)
2025-02-17 18:23:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858484416 ms)
2025-02-17 18:23:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858484416 ms)
2025-02-17 18:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:23:25.400182+00:00', 'runtime': '858480.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.45', 'adc_il_f': '0.246', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.7', 'ahc_t': '2652.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.086', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858484420 ms)
2025-02-17 18:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:22:25.400159+00:00', 'runtime': '858420.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.643', 'adc_vl_f': '14.493', 'adc_ic_f': '2.454', 'adc_il_f': '0.249', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.6', 'ahc_t': '2652.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.161', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.6', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858484421 ms)
2025-02-17 18:23:35.852 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858493672 ms)
2025-02-17 18:23:35.853 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858493672 ms)
2025-02-17 18:23:35.854 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:23:25.400028+00:00'} (T+858493673 ms)
2025-02-17 18:24:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858544413 ms)
2025-02-17 18:24:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858544414 ms)
2025-02-17 18:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:24:25.400244+00:00', 'runtime': '858540.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.464', 'adc_il_f': '0.29', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.7', 'ahc_t': '2652.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.527', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858544415 ms)
2025-02-17 18:24:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:23:25.400182+00:00', 'runtime': '858480.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.45', 'adc_il_f': '0.246', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.7', 'ahc_t': '2652.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.086', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858544415 ms)
2025-02-17 18:24:36.052 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858553871 ms)
2025-02-17 18:24:36.052 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858553872 ms)
2025-02-17 18:24:36.053 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:24:25.409122+00:00'} (T+858553873 ms)
2025-02-17 18:25:20.716 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+858598535 ms)
2025-02-17 18:25:20.716 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+858598536 ms)
2025-02-17 18:25:20.717 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52064)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858598536 ms)
2025-02-17 18:25:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858604413 ms)
2025-02-17 18:25:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858604414 ms)
2025-02-17 18:25:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:25:25.400256+00:00', 'runtime': '858600.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.454', 'adc_il_f': '0.285', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.7', 'ahc_t': '2652.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.146', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858604415 ms)
2025-02-17 18:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:24:25.400244+00:00', 'runtime': '858540.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.464', 'adc_il_f': '0.29', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.7', 'ahc_t': '2652.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.527', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858604416 ms)
2025-02-17 18:25:35.904 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858613723 ms)
2025-02-17 18:25:35.904 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858613724 ms)
2025-02-17 18:25:35.905 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:25:25.400280+00:00'} (T+858613725 ms)
2025-02-17 18:26:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858664413 ms)
2025-02-17 18:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858664413 ms)
2025-02-17 18:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:26:25.400164+00:00', 'runtime': '858660.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.425', 'adc_il_f': '0.263', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.8', 'ahc_t': '2652.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.602', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858664415 ms)
2025-02-17 18:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:25:25.400256+00:00', 'runtime': '858600.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.454', 'adc_il_f': '0.285', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.7', 'ahc_t': '2652.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.146', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.7', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858664416 ms)
2025-02-17 18:26:35.678 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858673497 ms)
2025-02-17 18:26:35.678 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858673497 ms)
2025-02-17 18:26:35.679 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:26:25.400410+00:00'} (T+858673498 ms)
2025-02-17 18:26:35.877 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+858673697 ms)
2025-02-17 18:26:35.878 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+858673697 ms)
2025-02-17 18:26:35.878 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52156)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858673698 ms)
2025-02-17 18:27:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858724411 ms)
2025-02-17 18:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858724411 ms)
2025-02-17 18:27:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:27:25.400148+00:00', 'runtime': '858720.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.433', 'adc_il_f': '0.275', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.8', 'ahc_t': '2652.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.527', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858724412 ms)
2025-02-17 18:27:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:26:25.400164+00:00', 'runtime': '858660.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.425', 'adc_il_f': '0.263', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.8', 'ahc_t': '2652.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.602', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858724413 ms)
2025-02-17 18:28:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858784411 ms)
2025-02-17 18:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858784411 ms)
2025-02-17 18:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:28:25.400160+00:00', 'runtime': '858780.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.401', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.9', 'ahc_t': '2652.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.316', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858784412 ms)
2025-02-17 18:28:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:27:25.400148+00:00', 'runtime': '858720.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.433', 'adc_il_f': '0.275', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.8', 'ahc_t': '2652.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.527', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.8', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858784413 ms)
2025-02-17 18:28:35.116 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+858792935 ms)
2025-02-17 18:28:35.116 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+858792935 ms)
2025-02-17 18:28:35.116 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52282)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+858792936 ms)
2025-02-17 18:29:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858844411 ms)
2025-02-17 18:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858844411 ms)
2025-02-17 18:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:29:25.400184+00:00', 'runtime': '858840.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.738', 'adc_vl_f': '14.493', 'adc_ic_f': '2.392', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.9', 'ahc_t': '2652.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.255', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858844412 ms)
2025-02-17 18:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:28:25.400160+00:00', 'runtime': '858780.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.401', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.9', 'ahc_t': '2652.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.316', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858844413 ms)
2025-02-17 18:29:35.884 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858853703 ms)
2025-02-17 18:29:35.884 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858853704 ms)
2025-02-17 18:29:35.885 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:29:25.400338+00:00'} (T+858853705 ms)
2025-02-17 18:30:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858904410 ms)
2025-02-17 18:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858904411 ms)
2025-02-17 18:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:30:25.400199+00:00', 'runtime': '858900.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.396', 'adc_il_f': '0.283', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.9', 'ahc_t': '2652.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.938', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858904412 ms)
2025-02-17 18:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:29:25.400184+00:00', 'runtime': '858840.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.738', 'adc_vl_f': '14.493', 'adc_ic_f': '2.392', 'adc_il_f': '0.287', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2652.9', 'ahc_t': '2652.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.255', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858904412 ms)
2025-02-17 18:30:35.788 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858913607 ms)
2025-02-17 18:30:35.788 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858913607 ms)
2025-02-17 18:30:35.789 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:30:25.400446+00:00'} (T+858913608 ms)
2025-02-17 18:31:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+858964409 ms)
2025-02-17 18:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+858964410 ms)
2025-02-17 18:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:31:25.400201+00:00', 'runtime': '858960.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.762', 'adc_vl_f': '14.493', 'adc_ic_f': '2.37', 'adc_il_f': '0.278', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.0', 'ahc_t': '2653.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.802', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858964411 ms)
2025-02-17 18:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:30:25.400199+00:00', 'runtime': '858900.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.396', 'adc_il_f': '0.283', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2652.9', 'ahc_t': '2652.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.938', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '22.9', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+858964411 ms)
2025-02-17 18:31:35.827 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858973647 ms)
2025-02-17 18:31:35.828 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+858973647 ms)
2025-02-17 18:31:35.829 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:31:25.400027+00:00'} (T+858973648 ms)
2025-02-17 18:32:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859024409 ms)
2025-02-17 18:32:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859024410 ms)
2025-02-17 18:32:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:32:25.400165+00:00', 'runtime': '859020.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.674', 'adc_vl_f': '14.493', 'adc_ic_f': '2.324', 'adc_il_f': '0.263', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.0', 'ahc_t': '2653.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.968', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859024410 ms)
2025-02-17 18:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:31:25.400201+00:00', 'runtime': '858960.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.762', 'adc_vl_f': '14.493', 'adc_ic_f': '2.37', 'adc_il_f': '0.278', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.0', 'ahc_t': '2653.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.802', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859024411 ms)
2025-02-17 18:32:35.756 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859033575 ms)
2025-02-17 18:32:35.757 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859033576 ms)
2025-02-17 18:32:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:32:25.399990+00:00'} (T+859033577 ms)
2025-02-17 18:33:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859084409 ms)
2025-02-17 18:33:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859084409 ms)
2025-02-17 18:33:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:33:25.400179+00:00', 'runtime': '859080.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.667', 'adc_vl_f': '14.493', 'adc_ic_f': '2.3', 'adc_il_f': '0.242', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.1', 'ahc_t': '2653.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '33.987', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859084410 ms)
2025-02-17 18:33:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:32:25.400165+00:00', 'runtime': '859020.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.674', 'adc_vl_f': '14.493', 'adc_ic_f': '2.324', 'adc_il_f': '0.263', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.0', 'ahc_t': '2653.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.968', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.0', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859084411 ms)
2025-02-17 18:33:35.855 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859093674 ms)
2025-02-17 18:33:35.856 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859093675 ms)
2025-02-17 18:33:35.857 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:33:25.400040+00:00'} (T+859093676 ms)
2025-02-17 18:33:43.500 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+859101319 ms)
2025-02-17 18:33:43.500 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+859101319 ms)
2025-02-17 18:33:43.501 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859101320 ms)
2025-02-17 18:33:43.501 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+859101321 ms)
2025-02-17 18:33:43.501 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+859101321 ms)
2025-02-17 18:33:43.502 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859101321 ms)
2025-02-17 18:33:53.557 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859111376 ms)
2025-02-17 18:33:53.557 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859111376 ms)
2025-02-17 18:33:53.558 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52624)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859111377 ms)
2025-02-17 18:34:17.588 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+859135408 ms)
2025-02-17 18:34:17.589 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+859135408 ms)
2025-02-17 18:34:17.589 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859135408 ms)
2025-02-17 18:34:17.590 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+859135409 ms)
2025-02-17 18:34:17.590 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+859135409 ms)
2025-02-17 18:34:17.590 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859135410 ms)
2025-02-17 18:34:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859144407 ms)
2025-02-17 18:34:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859144408 ms)
2025-02-17 18:34:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:34:25.400192+00:00', 'runtime': '859140.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.331', 'adc_il_f': '0.254', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.1', 'ahc_t': '2653.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.515', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859144409 ms)
2025-02-17 18:34:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:33:25.400179+00:00', 'runtime': '859080.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.667', 'adc_vl_f': '14.493', 'adc_ic_f': '2.3', 'adc_il_f': '0.242', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.1', 'ahc_t': '2653.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '33.987', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859144410 ms)
2025-02-17 18:34:29.649 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859147469 ms)
2025-02-17 18:34:29.650 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859147469 ms)
2025-02-17 18:34:29.650 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52662)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859147469 ms)
2025-02-17 18:34:35.766 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859153586 ms)
2025-02-17 18:34:35.767 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859153586 ms)
2025-02-17 18:34:35.768 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:34:25.400475+00:00'} (T+859153587 ms)
2025-02-17 18:35:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859204406 ms)
2025-02-17 18:35:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859204406 ms)
2025-02-17 18:35:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:35:25.400199+00:00', 'runtime': '859200.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.317', 'adc_il_f': '0.249', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.1', 'ahc_t': '2653.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.334', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859204407 ms)
2025-02-17 18:35:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:34:25.400192+00:00', 'runtime': '859140.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.331', 'adc_il_f': '0.254', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.1', 'ahc_t': '2653.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.515', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859204407 ms)
2025-02-17 18:35:33.774 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859211594 ms)
2025-02-17 18:35:33.775 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859211594 ms)
2025-02-17 18:35:33.775 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52734)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859211595 ms)
2025-02-17 18:35:35.678 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859213497 ms)
2025-02-17 18:35:35.678 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859213498 ms)
2025-02-17 18:35:35.679 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:35:25.400023+00:00'} (T+859213498 ms)
2025-02-17 18:36:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859264407 ms)
2025-02-17 18:36:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859264407 ms)
2025-02-17 18:36:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:36:25.400198+00:00', 'runtime': '859260.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.317', 'adc_il_f': '0.256', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.2', 'ahc_t': '2653.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.274', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859264408 ms)
2025-02-17 18:36:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:35:25.400199+00:00', 'runtime': '859200.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.317', 'adc_il_f': '0.249', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.1', 'ahc_t': '2653.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.334', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.1', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859264408 ms)
2025-02-17 18:36:35.666 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859273485 ms)
2025-02-17 18:36:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859273486 ms)
2025-02-17 18:36:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:36:25.400134+00:00'} (T+859273487 ms)
2025-02-17 18:37:17.970 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859315789 ms)
2025-02-17 18:37:17.970 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859315789 ms)
2025-02-17 18:37:17.971 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52856)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859315790 ms)
2025-02-17 18:37:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859324420 ms)
2025-02-17 18:37:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859324420 ms)
2025-02-17 18:37:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:37:25.400164+00:00', 'runtime': '859320.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.336', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.2', 'ahc_t': '2653.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.123', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859324421 ms)
2025-02-17 18:37:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:36:25.400198+00:00', 'runtime': '859260.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.317', 'adc_il_f': '0.256', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.2', 'ahc_t': '2653.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.274', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859324421 ms)
2025-02-17 18:37:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859333429 ms)
2025-02-17 18:37:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859333429 ms)
2025-02-17 18:37:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:37:25.400130+00:00'} (T+859333430 ms)
2025-02-17 18:37:56.017 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+859353836 ms)
2025-02-17 18:37:56.017 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+859353836 ms)
2025-02-17 18:37:56.017 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859353837 ms)
2025-02-17 18:37:56.018 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+859353837 ms)
2025-02-17 18:37:56.018 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+859353838 ms)
2025-02-17 18:37:56.019 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859353838 ms)
2025-02-17 18:38:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859384419 ms)
2025-02-17 18:38:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859384420 ms)
2025-02-17 18:38:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:38:25.400167+00:00', 'runtime': '859380.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.652', 'adc_vl_f': '14.493', 'adc_ic_f': '2.334', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.3', 'ahc_t': '2653.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '33.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859384421 ms)
2025-02-17 18:38:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:37:25.400164+00:00', 'runtime': '859320.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.336', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.2', 'ahc_t': '2653.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '34.123', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.2', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859384421 ms)
2025-02-17 18:38:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859393437 ms)
2025-02-17 18:38:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859393437 ms)
2025-02-17 18:38:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:38:25.400187+00:00'} (T+859393438 ms)
2025-02-17 18:39:22.216 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859440035 ms)
2025-02-17 18:39:22.216 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859440035 ms)
2025-02-17 18:39:22.216 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52990)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859440036 ms)
2025-02-17 18:39:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859444402 ms)
2025-02-17 18:39:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859444403 ms)
2025-02-17 18:39:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:39:25.400179+00:00', 'runtime': '859440.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.496', 'adc_ic_f': '2.198', 'adc_il_f': '0.36', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.3', 'ahc_t': '2653.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.13', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859444404 ms)
2025-02-17 18:39:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:38:25.400167+00:00', 'runtime': '859380.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.652', 'adc_vl_f': '14.493', 'adc_ic_f': '2.334', 'adc_il_f': '0.292', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.3', 'ahc_t': '2653.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '33.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859444404 ms)
2025-02-17 18:39:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859453430 ms)
2025-02-17 18:39:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859453430 ms)
2025-02-17 18:39:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:39:25.400203+00:00'} (T+859453432 ms)
2025-02-17 18:40:24.289 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+859502109 ms)
2025-02-17 18:40:24.290 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+859502109 ms)
2025-02-17 18:40:24.292 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859502111 ms)
2025-02-17 18:40:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859504417 ms)
2025-02-17 18:40:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859504418 ms)
2025-02-17 18:40:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:40:25.400200+00:00', 'runtime': '859500.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.774', 'adc_vl_f': '14.496', 'adc_ic_f': '2.167', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.3', 'ahc_t': '2653.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.311', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.3', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859504419 ms)
2025-02-17 18:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:39:25.400179+00:00', 'runtime': '859440.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.496', 'adc_ic_f': '2.198', 'adc_il_f': '0.36', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.3', 'ahc_t': '2653.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.4', 'ahl_t': '1591.4', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.13', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.3', 'ahl_daily': '7.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859504419 ms)
2025-02-17 18:40:35.694 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859513513 ms)
2025-02-17 18:40:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859513513 ms)
2025-02-17 18:40:35.695 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:40:25.400204+00:00'} (T+859513514 ms)
2025-02-17 18:40:51.397 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859529217 ms)
2025-02-17 18:40:51.398 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859529217 ms)
2025-02-17 18:40:51.398 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53088)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859529217 ms)
2025-02-17 18:41:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859564416 ms)
2025-02-17 18:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859564416 ms)
2025-02-17 18:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:41:25.400191+00:00', 'runtime': '859560.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.86', 'adc_vl_f': '14.493', 'adc_ic_f': '2.133', 'adc_il_f': '0.321', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.4', 'ahc_t': '2653.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.994', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.4', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859564417 ms)
2025-02-17 18:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:40:25.400200+00:00', 'runtime': '859500.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.774', 'adc_vl_f': '14.496', 'adc_ic_f': '2.167', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.3', 'ahc_t': '2653.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.311', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.3', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859564418 ms)
2025-02-17 18:41:35.905 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859573724 ms)
2025-02-17 18:41:35.906 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859573725 ms)
2025-02-17 18:41:35.907 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:41:25.400159+00:00'} (T+859573726 ms)
2025-02-17 18:42:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859624415 ms)
2025-02-17 18:42:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859624416 ms)
2025-02-17 18:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:42:25.402871+00:00', 'runtime': '859620.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.145', 'adc_il_f': '0.331', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.4', 'ahc_t': '2653.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.752', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.4', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859624418 ms)
2025-02-17 18:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:41:25.400191+00:00', 'runtime': '859560.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.86', 'adc_vl_f': '14.493', 'adc_ic_f': '2.133', 'adc_il_f': '0.321', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.4', 'ahc_t': '2653.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.994', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.4', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859624418 ms)
2025-02-17 18:42:32.607 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859630426 ms)
2025-02-17 18:42:32.607 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859630426 ms)
2025-02-17 18:42:32.607 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53194)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859630427 ms)
2025-02-17 18:42:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859633435 ms)
2025-02-17 18:42:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859633435 ms)
2025-02-17 18:42:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:42:25.400273+00:00'} (T+859633436 ms)
2025-02-17 18:43:23.737 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859681557 ms)
2025-02-17 18:43:23.738 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859681557 ms)
2025-02-17 18:43:23.738 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53250)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859681558 ms)
2025-02-17 18:43:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859684415 ms)
2025-02-17 18:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859684415 ms)
2025-02-17 18:43:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:43:25.402786+00:00', 'runtime': '859680.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.128', 'adc_il_f': '0.321', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.4', 'ahc_t': '2653.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.813', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.4', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859684416 ms)
2025-02-17 18:43:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:42:25.402871+00:00', 'runtime': '859620.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.145', 'adc_il_f': '0.331', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.4', 'ahc_t': '2653.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.752', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.4', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859684417 ms)
2025-02-17 18:43:35.822 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859693641 ms)
2025-02-17 18:43:35.823 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859693642 ms)
2025-02-17 18:43:35.824 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:43:25.400180+00:00'} (T+859693643 ms)
2025-02-17 18:43:54.854 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859712674 ms)
2025-02-17 18:43:54.855 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859712674 ms)
2025-02-17 18:43:54.855 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53286)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859712674 ms)
2025-02-17 18:44:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859744412 ms)
2025-02-17 18:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859744413 ms)
2025-02-17 18:44:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:44:25.402594+00:00', 'runtime': '859740.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.735', 'adc_vl_f': '14.493', 'adc_ic_f': '2.136', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.5', 'ahc_t': '2653.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.677', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.5', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859744414 ms)
2025-02-17 18:44:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:43:25.402786+00:00', 'runtime': '859680.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '2.128', 'adc_il_f': '0.321', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.4', 'ahc_t': '2653.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.813', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.4', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859744414 ms)
2025-02-17 18:45:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859804414 ms)
2025-02-17 18:45:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859804414 ms)
2025-02-17 18:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:45:25.400121+00:00', 'runtime': '859800.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.496', 'adc_ic_f': '2.148', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.5', 'ahc_t': '2653.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.575', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.5', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859804415 ms)
2025-02-17 18:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:44:25.402594+00:00', 'runtime': '859740.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.735', 'adc_vl_f': '14.493', 'adc_ic_f': '2.136', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.5', 'ahc_t': '2653.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.677', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.5', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859804416 ms)
2025-02-17 18:45:53.067 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859830887 ms)
2025-02-17 18:45:53.068 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859830887 ms)
2025-02-17 18:45:53.068 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53422)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859830887 ms)
2025-02-17 18:46:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859864411 ms)
2025-02-17 18:46:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859864412 ms)
2025-02-17 18:46:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:46:25.400181+00:00', 'runtime': '859860.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.496', 'adc_ic_f': '2.138', 'adc_il_f': '0.355', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.5', 'ahc_t': '2653.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.148', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.5', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859864413 ms)
2025-02-17 18:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:45:25.400121+00:00', 'runtime': '859800.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.664', 'adc_vl_f': '14.496', 'adc_ic_f': '2.148', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.5', 'ahc_t': '2653.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.575', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.5', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859864413 ms)
2025-02-17 18:46:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859873437 ms)
2025-02-17 18:46:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859873437 ms)
2025-02-17 18:46:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:46:25.413072+00:00'} (T+859873438 ms)
2025-02-17 18:47:07.236 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859905056 ms)
2025-02-17 18:47:07.237 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859905056 ms)
2025-02-17 18:47:07.237 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53502)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859905056 ms)
2025-02-17 18:47:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859924411 ms)
2025-02-17 18:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859924411 ms)
2025-02-17 18:47:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:47:25.400189+00:00', 'runtime': '859920.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.777', 'adc_vl_f': '14.493', 'adc_ic_f': '2.14', 'adc_il_f': '0.365', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.6', 'ahc_t': '2653.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.345', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.6', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859924412 ms)
2025-02-17 18:47:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:46:25.400181+00:00', 'runtime': '859860.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.496', 'adc_ic_f': '2.138', 'adc_il_f': '0.355', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.5', 'ahc_t': '2653.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.148', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.5', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859924413 ms)
2025-02-17 18:47:35.763 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859933582 ms)
2025-02-17 18:47:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859933583 ms)
2025-02-17 18:47:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:47:25.400203+00:00'} (T+859933584 ms)
2025-02-17 18:48:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+859984413 ms)
2025-02-17 18:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+859984413 ms)
2025-02-17 18:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:48:25.400179+00:00', 'runtime': '859980.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.729', 'adc_vl_f': '14.493', 'adc_ic_f': '2.099', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.6', 'ahc_t': '2653.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.043', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.6', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859984414 ms)
2025-02-17 18:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:47:25.400189+00:00', 'runtime': '859920.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.777', 'adc_vl_f': '14.493', 'adc_ic_f': '2.14', 'adc_il_f': '0.365', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2653.6', 'ahc_t': '2653.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.345', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.6', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+859984415 ms)
2025-02-17 18:48:31.417 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+859989236 ms)
2025-02-17 18:48:31.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+859989236 ms)
2025-02-17 18:48:31.418 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53592)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+859989237 ms)
2025-02-17 18:48:35.734 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859993553 ms)
2025-02-17 18:48:35.735 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+859993554 ms)
2025-02-17 18:48:35.735 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:48:25.400033+00:00'} (T+859993555 ms)
2025-02-17 18:49:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860044412 ms)
2025-02-17 18:49:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860044413 ms)
2025-02-17 18:49:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:49:25.400167+00:00', 'runtime': '860040.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.661', 'adc_vl_f': '14.493', 'adc_ic_f': '2.056', 'adc_il_f': '0.314', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.6', 'ahc_t': '2653.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.786', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.6', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860044413 ms)
2025-02-17 18:49:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:48:25.400179+00:00', 'runtime': '859980.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.729', 'adc_vl_f': '14.493', 'adc_ic_f': '2.099', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.6', 'ahc_t': '2653.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.043', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.6', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860044414 ms)
2025-02-17 18:50:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860104412 ms)
2025-02-17 18:50:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860104412 ms)
2025-02-17 18:50:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:50:25.400166+00:00', 'runtime': '860100.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.57', 'adc_vl_f': '14.493', 'adc_ic_f': '2.075', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.7', 'ahc_t': '2653.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.684', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.7', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860104414 ms)
2025-02-17 18:50:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:49:25.400167+00:00', 'runtime': '860040.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.661', 'adc_vl_f': '14.493', 'adc_ic_f': '2.056', 'adc_il_f': '0.314', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.6', 'ahc_t': '2653.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.786', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.6', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860104414 ms)
2025-02-17 18:51:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860164412 ms)
2025-02-17 18:51:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860164412 ms)
2025-02-17 18:51:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:51:25.400160+00:00', 'runtime': '860160.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.555', 'adc_vl_f': '14.493', 'adc_ic_f': '2.044', 'adc_il_f': '0.316', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.7', 'ahc_t': '2653.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.62', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.7', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860164413 ms)
2025-02-17 18:51:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:50:25.400166+00:00', 'runtime': '860100.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.57', 'adc_vl_f': '14.493', 'adc_ic_f': '2.075', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.7', 'ahc_t': '2653.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.684', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.7', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860164413 ms)
2025-02-17 18:51:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860173426 ms)
2025-02-17 18:51:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860173427 ms)
2025-02-17 18:51:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:51:25.400336+00:00'} (T+860173435 ms)
2025-02-17 18:52:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860224411 ms)
2025-02-17 18:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860224411 ms)
2025-02-17 18:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:52:25.400184+00:00', 'runtime': '860220.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.558', 'adc_vl_f': '14.493', 'adc_ic_f': '2.058', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.8', 'ahc_t': '2653.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.59', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.8', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860224412 ms)
2025-02-17 18:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:51:25.400160+00:00', 'runtime': '860160.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.555', 'adc_vl_f': '14.493', 'adc_ic_f': '2.044', 'adc_il_f': '0.316', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.7', 'ahc_t': '2653.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.62', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.7', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860224413 ms)
2025-02-17 18:52:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860233427 ms)
2025-02-17 18:52:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860233428 ms)
2025-02-17 18:52:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:52:25.400208+00:00'} (T+860233432 ms)
2025-02-17 18:53:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860284411 ms)
2025-02-17 18:53:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860284411 ms)
2025-02-17 18:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:53:25.400181+00:00', 'runtime': '860280.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.082', 'adc_il_f': '0.374', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.8', 'ahc_t': '2653.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.695', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.8', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860284412 ms)
2025-02-17 18:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:52:25.400184+00:00', 'runtime': '860220.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.558', 'adc_vl_f': '14.493', 'adc_ic_f': '2.058', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.8', 'ahc_t': '2653.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.59', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.8', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860284413 ms)
2025-02-17 18:53:35.908 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860293727 ms)
2025-02-17 18:53:35.908 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860293728 ms)
2025-02-17 18:53:35.910 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:53:25.400549+00:00'} (T+860293729 ms)
2025-02-17 18:54:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860344410 ms)
2025-02-17 18:54:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860344411 ms)
2025-02-17 18:54:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:54:25.400167+00:00', 'runtime': '860340.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.765', 'adc_vl_f': '14.493', 'adc_ic_f': '2.056', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.8', 'ahc_t': '2653.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.333', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.8', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860344412 ms)
2025-02-17 18:54:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:53:25.400181+00:00', 'runtime': '860280.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.664', 'adc_vl_f': '14.493', 'adc_ic_f': '2.082', 'adc_il_f': '0.374', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.8', 'ahc_t': '2653.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20125', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.695', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.8', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860344412 ms)
2025-02-17 18:55:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860404408 ms)
2025-02-17 18:55:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860404409 ms)
2025-02-17 18:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:55:25.400178+00:00', 'runtime': '860400.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.848', 'adc_vl_f': '14.493', 'adc_ic_f': '2.063', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.9', 'ahc_t': '2653.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.427', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.9', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860404410 ms)
2025-02-17 18:55:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:54:25.400167+00:00', 'runtime': '860340.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.765', 'adc_vl_f': '14.493', 'adc_ic_f': '2.056', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.8', 'ahc_t': '2653.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.333', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.8', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860404410 ms)
2025-02-17 18:56:07.067 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+860444887 ms)
2025-02-17 18:56:07.068 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+860444887 ms)
2025-02-17 18:56:07.068 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54086)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+860444887 ms)
2025-02-17 18:56:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860464407 ms)
2025-02-17 18:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860464408 ms)
2025-02-17 18:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:56:25.400183+00:00', 'runtime': '860460.001', 'adc_vb_f': '14.514', 'adc_va_f': '21.884', 'adc_vl_f': '14.493', 'adc_ic_f': '2.056', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.9', 'ahc_t': '2653.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.367', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.9', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860464409 ms)
2025-02-17 18:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:55:25.400178+00:00', 'runtime': '860400.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.848', 'adc_vl_f': '14.493', 'adc_ic_f': '2.063', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.9', 'ahc_t': '2653.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.427', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.9', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860464409 ms)
2025-02-17 18:57:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860524408 ms)
2025-02-17 18:57:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860524408 ms)
2025-02-17 18:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:57:25.400191+00:00', 'runtime': '860520.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.034', 'adc_il_f': '0.35', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.9', 'ahc_t': '2653.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.107', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.9', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860524409 ms)
2025-02-17 18:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:56:25.400183+00:00', 'runtime': '860460.001', 'adc_vb_f': '14.514', 'adc_va_f': '21.884', 'adc_vl_f': '14.493', 'adc_ic_f': '2.056', 'adc_il_f': '0.343', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.9', 'ahc_t': '2653.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.367', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.9', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860524410 ms)
2025-02-17 18:57:35.619 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860533439 ms)
2025-02-17 18:57:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860533439 ms)
2025-02-17 18:57:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:57:25.400350+00:00'} (T+860533446 ms)
2025-02-17 18:58:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860584405 ms)
2025-02-17 18:58:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860584405 ms)
2025-02-17 18:58:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:58:25.400207+00:00', 'runtime': '860580.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.046', 'adc_il_f': '0.353', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.0', 'ahc_t': '2654.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.895', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.0', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860584406 ms)
2025-02-17 18:58:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:57:25.400191+00:00', 'runtime': '860520.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.034', 'adc_il_f': '0.35', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2653.9', 'ahc_t': '2653.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.107', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '23.9', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860584406 ms)
2025-02-17 18:58:35.622 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860593441 ms)
2025-02-17 18:58:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860593442 ms)
2025-02-17 18:58:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:58:25.400209+00:00'} (T+860593449 ms)
2025-02-17 18:59:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860644404 ms)
2025-02-17 18:59:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860644405 ms)
2025-02-17 18:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 18:59:25.400185+00:00', 'runtime': '860640.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.939', 'adc_vl_f': '14.493', 'adc_ic_f': '2.027', 'adc_il_f': '0.35', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.0', 'ahc_t': '2654.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.925', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.0', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860644406 ms)
2025-02-17 18:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:58:25.400207+00:00', 'runtime': '860580.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '2.046', 'adc_il_f': '0.353', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.0', 'ahc_t': '2654.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.5', 'ahl_t': '1591.5', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.895', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.0', 'ahl_daily': '7.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860644406 ms)
2025-02-17 18:59:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860653433 ms)
2025-02-17 18:59:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860653433 ms)
2025-02-17 18:59:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 18:59:25.400197+00:00'} (T+860653440 ms)
2025-02-17 18:59:49.397 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+860667216 ms)
2025-02-17 18:59:49.397 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+860667217 ms)
2025-02-17 18:59:49.398 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54324)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+860667217 ms)
2025-02-17 19:00:25.489 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+860703309 ms)
2025-02-17 19:00:25.490 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+860703309 ms)
2025-02-17 19:00:25.490 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54362)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+860703309 ms)
2025-02-17 19:00:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860704404 ms)
2025-02-17 19:00:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860704404 ms)
2025-02-17 19:00:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:00:25.400183+00:00', 'runtime': '860700.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.979', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.0', 'ahc_t': '2654.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.744', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.0', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860704405 ms)
2025-02-17 19:00:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 18:59:25.400185+00:00', 'runtime': '860640.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.939', 'adc_vl_f': '14.493', 'adc_ic_f': '2.027', 'adc_il_f': '0.35', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.0', 'ahc_t': '2654.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.925', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.0', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860704406 ms)
2025-02-17 19:00:35.756 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860713575 ms)
2025-02-17 19:00:35.756 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860713575 ms)
2025-02-17 19:00:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:00:25.400183+00:00'} (T+860713577 ms)
2025-02-17 19:01:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860764418 ms)
2025-02-17 19:01:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860764419 ms)
2025-02-17 19:01:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:01:25.402666+00:00', 'runtime': '860760.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.872', 'adc_vl_f': '14.493', 'adc_ic_f': '1.962', 'adc_il_f': '0.316', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.1', 'ahc_t': '2654.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.608', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.1', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860764420 ms)
2025-02-17 19:01:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:00:25.400183+00:00', 'runtime': '860700.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.979', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.0', 'ahc_t': '2654.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.744', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.0', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860764420 ms)
2025-02-17 19:01:35.578 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+860773397 ms)
2025-02-17 19:01:35.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+860773398 ms)
2025-02-17 19:01:35.579 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+860773398 ms)
2025-02-17 19:01:35.778 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860773598 ms)
2025-02-17 19:01:35.779 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860773598 ms)
2025-02-17 19:01:35.780 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:01:25.400029+00:00'} (T+860773599 ms)
2025-02-17 19:02:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860824416 ms)
2025-02-17 19:02:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860824417 ms)
2025-02-17 19:02:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:02:25.400007+00:00', 'runtime': '860820.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '1.971', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.1', 'ahc_t': '2654.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.367', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.1', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860824418 ms)
2025-02-17 19:02:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:01:25.402666+00:00', 'runtime': '860760.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.872', 'adc_vl_f': '14.493', 'adc_ic_f': '1.962', 'adc_il_f': '0.316', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.1', 'ahc_t': '2654.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.608', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.1', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860824418 ms)
2025-02-17 19:02:35.759 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860833578 ms)
2025-02-17 19:02:35.760 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+860833579 ms)
2025-02-17 19:02:35.761 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:02:25.409122+00:00'} (T+860833580 ms)
2025-02-17 19:03:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860884415 ms)
2025-02-17 19:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860884415 ms)
2025-02-17 19:03:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:03:25.402642+00:00', 'runtime': '860880.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '1.974', 'adc_il_f': '0.324', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.1', 'ahc_t': '2654.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.503', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.1', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860884416 ms)
2025-02-17 19:03:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:02:25.400007+00:00', 'runtime': '860820.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '1.971', 'adc_il_f': '0.319', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.1', 'ahc_t': '2654.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.367', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.1', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860884417 ms)
2025-02-17 19:04:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+860944417 ms)
2025-02-17 19:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+860944417 ms)
2025-02-17 19:04:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:04:25.400064+00:00', 'runtime': '860940.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.786', 'adc_vl_f': '14.493', 'adc_ic_f': '1.993', 'adc_il_f': '0.331', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.2', 'ahc_t': '2654.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.174', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.2', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860944418 ms)
2025-02-17 19:04:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:03:25.402642+00:00', 'runtime': '860880.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.771', 'adc_vl_f': '14.493', 'adc_ic_f': '1.974', 'adc_il_f': '0.324', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.1', 'ahc_t': '2654.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.503', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.1', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+860944419 ms)
2025-02-17 19:05:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861004416 ms)
2025-02-17 19:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861004417 ms)
2025-02-17 19:05:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:05:25.402793+00:00', 'runtime': '861000.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.005', 'adc_il_f': '0.36', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.2', 'ahc_t': '2654.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.08', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.2', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861004418 ms)
2025-02-17 19:05:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:04:25.400064+00:00', 'runtime': '860940.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.786', 'adc_vl_f': '14.493', 'adc_ic_f': '1.993', 'adc_il_f': '0.331', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.2', 'ahc_t': '2654.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.174', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.2', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861004418 ms)
2025-02-17 19:06:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861064416 ms)
2025-02-17 19:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861064416 ms)
2025-02-17 19:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:06:25.400171+00:00', 'runtime': '861060.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.986', 'adc_il_f': '0.355', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.2', 'ahc_t': '2654.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.989', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.2', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861064417 ms)
2025-02-17 19:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:05:25.402793+00:00', 'runtime': '861000.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '2.005', 'adc_il_f': '0.36', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.2', 'ahc_t': '2654.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.08', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.2', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861064418 ms)
2025-02-17 19:07:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861124417 ms)
2025-02-17 19:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861124418 ms)
2025-02-17 19:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:07:25.400175+00:00', 'runtime': '861120.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.814', 'adc_vl_f': '14.493', 'adc_ic_f': '1.991', 'adc_il_f': '0.358', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.3', 'ahc_t': '2654.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.472', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.3', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861124419 ms)
2025-02-17 19:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:06:25.400171+00:00', 'runtime': '861060.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.986', 'adc_il_f': '0.355', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.2', 'ahc_t': '2654.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.989', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.2', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861124420 ms)
2025-02-17 19:07:35.674 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861133493 ms)
2025-02-17 19:07:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861133494 ms)
2025-02-17 19:07:35.675 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:07:25.400323+00:00'} (T+861133495 ms)
2025-02-17 19:08:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861184415 ms)
2025-02-17 19:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861184416 ms)
2025-02-17 19:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:08:25.400175+00:00', 'runtime': '861180.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.82', 'adc_vl_f': '14.493', 'adc_ic_f': '1.983', 'adc_il_f': '0.367', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.3', 'ahc_t': '2654.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.838', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.3', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861184417 ms)
2025-02-17 19:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:07:25.400175+00:00', 'runtime': '861120.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.814', 'adc_vl_f': '14.493', 'adc_ic_f': '1.991', 'adc_il_f': '0.358', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.3', 'ahc_t': '2654.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.472', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.3', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861184417 ms)
2025-02-17 19:08:57.207 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+861215026 ms)
2025-02-17 19:08:57.207 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+861215026 ms)
2025-02-17 19:08:57.208 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54916)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861215027 ms)
2025-02-17 19:09:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861244413 ms)
2025-02-17 19:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861244413 ms)
2025-02-17 19:09:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:09:25.400180+00:00', 'runtime': '861240.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.962', 'adc_il_f': '0.333', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2654.3', 'ahc_t': '2654.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.963', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.3', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861244414 ms)
2025-02-17 19:09:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:08:25.400175+00:00', 'runtime': '861180.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.82', 'adc_vl_f': '14.493', 'adc_ic_f': '1.983', 'adc_il_f': '0.367', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.3', 'ahc_t': '2654.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.838', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.3', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861244415 ms)
2025-02-17 19:10:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861304413 ms)
2025-02-17 19:10:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861304413 ms)
2025-02-17 19:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:10:25.400166+00:00', 'runtime': '861300.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.952', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.4', 'ahc_t': '2654.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.733', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.4', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861304414 ms)
2025-02-17 19:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:09:25.400180+00:00', 'runtime': '861240.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.962', 'adc_il_f': '0.333', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.508', 'vb_ref': '14.03', 'ahc_r': '2654.3', 'ahc_t': '2654.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.963', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.3', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861304415 ms)
2025-02-17 19:10:35.756 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861313575 ms)
2025-02-17 19:10:35.757 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861313576 ms)
2025-02-17 19:10:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:10:25.400339+00:00'} (T+861313578 ms)
2025-02-17 19:11:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861364412 ms)
2025-02-17 19:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861364412 ms)
2025-02-17 19:11:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:11:25.400208+00:00', 'runtime': '861360.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.93', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.4', 'ahc_t': '2654.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.808', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.4', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861364413 ms)
2025-02-17 19:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:10:25.400166+00:00', 'runtime': '861300.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.952', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.4', 'ahc_t': '2654.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.733', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.4', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861364414 ms)
2025-02-17 19:12:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861424409 ms)
2025-02-17 19:12:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861424409 ms)
2025-02-17 19:12:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:12:25.400176+00:00', 'runtime': '861420.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.924', 'adc_vl_f': '14.493', 'adc_ic_f': '1.899', 'adc_il_f': '0.302', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.4', 'ahc_t': '2654.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.812', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.4', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861424410 ms)
2025-02-17 19:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:11:25.400208+00:00', 'runtime': '861360.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.93', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.4', 'ahc_t': '2654.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.808', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.4', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861424411 ms)
2025-02-17 19:13:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861484410 ms)
2025-02-17 19:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861484410 ms)
2025-02-17 19:13:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:13:25.400203+00:00', 'runtime': '861480.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.924', 'adc_vl_f': '14.493', 'adc_ic_f': '1.911', 'adc_il_f': '0.324', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.491', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861484411 ms)
2025-02-17 19:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:12:25.400176+00:00', 'runtime': '861420.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.924', 'adc_vl_f': '14.493', 'adc_ic_f': '1.899', 'adc_il_f': '0.302', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.4', 'ahc_t': '2654.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.812', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.4', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861484412 ms)
2025-02-17 19:14:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861544409 ms)
2025-02-17 19:14:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861544410 ms)
2025-02-17 19:14:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:14:25.400164+00:00', 'runtime': '861540.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.899', 'adc_vl_f': '14.496', 'adc_ic_f': '1.892', 'adc_il_f': '0.304', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.404', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861544411 ms)
2025-02-17 19:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:13:25.400203+00:00', 'runtime': '861480.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.924', 'adc_vl_f': '14.493', 'adc_ic_f': '1.911', 'adc_il_f': '0.324', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.491', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861544411 ms)
2025-02-17 19:15:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861604410 ms)
2025-02-17 19:15:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861604411 ms)
2025-02-17 19:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:15:25.400204+00:00', 'runtime': '861600.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.964', 'adc_vl_f': '14.493', 'adc_ic_f': '1.901', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.234', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861604412 ms)
2025-02-17 19:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:14:25.400164+00:00', 'runtime': '861540.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.899', 'adc_vl_f': '14.496', 'adc_ic_f': '1.892', 'adc_il_f': '0.304', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.404', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861604412 ms)
2025-02-17 19:15:31.780 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+861609599 ms)
2025-02-17 19:15:31.780 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+861609599 ms)
2025-02-17 19:15:31.780 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55330)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861609600 ms)
2025-02-17 19:15:35.683 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861613502 ms)
2025-02-17 19:15:35.683 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861613503 ms)
2025-02-17 19:15:35.684 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:15:25.400346+00:00'} (T+861613504 ms)
2025-02-17 19:16:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861664411 ms)
2025-02-17 19:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861664411 ms)
2025-02-17 19:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:16:25.400164+00:00', 'runtime': '861660.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.904', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.446', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861664412 ms)
2025-02-17 19:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:15:25.400204+00:00', 'runtime': '861600.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.964', 'adc_vl_f': '14.493', 'adc_ic_f': '1.901', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.234', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861664414 ms)
2025-02-17 19:16:35.674 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861673493 ms)
2025-02-17 19:16:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861673494 ms)
2025-02-17 19:16:35.675 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:16:25.400102+00:00'} (T+861673495 ms)
2025-02-17 19:17:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861724408 ms)
2025-02-17 19:17:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861724409 ms)
2025-02-17 19:17:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:17:25.400177+00:00', 'runtime': '861720.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '1.892', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.6', 'ahc_t': '2654.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.963', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.6', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861724410 ms)
2025-02-17 19:17:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:16:25.400164+00:00', 'runtime': '861660.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.904', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.5', 'ahc_t': '2654.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.446', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.5', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861724410 ms)
2025-02-17 19:17:33.929 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731749 ms)
2025-02-17 19:17:33.930 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731749 ms)
2025-02-17 19:17:33.930 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731749 ms)
2025-02-17 19:17:33.931 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731750 ms)
2025-02-17 19:17:33.931 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731750 ms)
2025-02-17 19:17:33.931 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731750 ms)
2025-02-17 19:17:33.931 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731751 ms)
2025-02-17 19:17:33.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731751 ms)
2025-02-17 19:17:33.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731751 ms)
2025-02-17 19:17:33.932 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731751 ms)
2025-02-17 19:17:33.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731752 ms)
2025-02-17 19:17:33.933 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731752 ms)
2025-02-17 19:17:33.933 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731752 ms)
2025-02-17 19:17:33.933 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731752 ms)
2025-02-17 19:17:33.933 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731753 ms)
2025-02-17 19:17:33.934 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731753 ms)
2025-02-17 19:17:33.934 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731753 ms)
2025-02-17 19:17:33.934 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731754 ms)
2025-02-17 19:17:33.935 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731754 ms)
2025-02-17 19:17:33.935 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731754 ms)
2025-02-17 19:17:33.935 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731754 ms)
2025-02-17 19:17:33.935 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731755 ms)
2025-02-17 19:17:33.936 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731755 ms)
2025-02-17 19:17:33.936 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731755 ms)
2025-02-17 19:17:33.936 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861731756 ms)
2025-02-17 19:17:33.936 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861731756 ms)
2025-02-17 19:17:33.937 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861731756 ms)
2025-02-17 19:17:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861733429 ms)
2025-02-17 19:17:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+861733430 ms)
2025-02-17 19:17:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:17:25.400116+00:00'} (T+861733431 ms)
2025-02-17 19:18:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861784408 ms)
2025-02-17 19:18:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861784409 ms)
2025-02-17 19:18:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:18:25.400194+00:00', 'runtime': '861780.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.982', 'adc_vl_f': '14.493', 'adc_ic_f': '1.937', 'adc_il_f': '0.365', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.6', 'ahc_t': '2654.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.234', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.6', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861784410 ms)
2025-02-17 19:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:17:25.400177+00:00', 'runtime': '861720.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.988', 'adc_vl_f': '14.493', 'adc_ic_f': '1.892', 'adc_il_f': '0.329', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.6', 'ahc_t': '2654.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.6', 'ahl_t': '1591.6', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.963', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.6', 'ahl_daily': '7.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861784410 ms)
2025-02-17 19:18:36.028 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861793847 ms)
2025-02-17 19:18:36.028 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861793847 ms)
2025-02-17 19:18:36.029 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861793848 ms)
2025-02-17 19:18:36.030 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861793849 ms)
2025-02-17 19:18:36.030 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861793849 ms)
2025-02-17 19:18:36.031 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861793850 ms)
2025-02-17 19:18:36.032 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+861793852 ms)
2025-02-17 19:18:36.033 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+861793852 ms)
2025-02-17 19:18:36.034 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861793853 ms)
2025-02-17 19:19:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861844409 ms)
2025-02-17 19:19:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861844409 ms)
2025-02-17 19:19:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:19:25.400205+00:00', 'runtime': '861840.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.939', 'adc_vl_f': '14.493', 'adc_ic_f': '1.923', 'adc_il_f': '0.367', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.6', 'ahc_t': '2654.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.374', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.6', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861844410 ms)
2025-02-17 19:19:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:18:25.400194+00:00', 'runtime': '861780.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.982', 'adc_vl_f': '14.493', 'adc_ic_f': '1.937', 'adc_il_f': '0.365', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.6', 'ahc_t': '2654.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.234', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.6', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861844410 ms)
2025-02-17 19:20:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861904406 ms)
2025-02-17 19:20:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861904406 ms)
2025-02-17 19:20:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:20:25.400581+00:00', 'runtime': '861900.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.881', 'adc_vl_f': '14.493', 'adc_ic_f': '1.918', 'adc_il_f': '0.36', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.7', 'ahc_t': '2654.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.404', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.7', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861904407 ms)
2025-02-17 19:20:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:19:25.400205+00:00', 'runtime': '861840.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.939', 'adc_vl_f': '14.493', 'adc_ic_f': '1.923', 'adc_il_f': '0.367', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.6', 'ahc_t': '2654.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.374', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.6', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861904408 ms)
2025-02-17 19:20:45.253 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+861923072 ms)
2025-02-17 19:20:45.253 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+861923072 ms)
2025-02-17 19:20:45.253 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55672)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+861923073 ms)
2025-02-17 19:21:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+861964407 ms)
2025-02-17 19:21:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+861964408 ms)
2025-02-17 19:21:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:21:25.402763+00:00', 'runtime': '861960.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.887', 'adc_vl_f': '14.493', 'adc_ic_f': '1.894', 'adc_il_f': '0.353', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.7', 'ahc_t': '2654.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.887', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.7', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861964409 ms)
2025-02-17 19:21:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:20:25.400581+00:00', 'runtime': '861900.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.881', 'adc_vl_f': '14.493', 'adc_ic_f': '1.918', 'adc_il_f': '0.36', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.7', 'ahc_t': '2654.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.404', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.7', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+861964409 ms)
2025-02-17 19:22:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862024407 ms)
2025-02-17 19:22:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862024407 ms)
2025-02-17 19:22:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:22:25.400181+00:00', 'runtime': '862020.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.866', 'adc_vl_f': '14.493', 'adc_ic_f': '1.846', 'adc_il_f': '0.333', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.7', 'ahc_t': '2654.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.963', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.7', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862024408 ms)
2025-02-17 19:22:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:21:25.402763+00:00', 'runtime': '861960.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.887', 'adc_vl_f': '14.493', 'adc_ic_f': '1.894', 'adc_il_f': '0.353', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.7', 'ahc_t': '2654.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.887', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.7', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862024409 ms)
2025-02-17 19:23:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862084406 ms)
2025-02-17 19:23:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862084406 ms)
2025-02-17 19:23:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:23:25.400206+00:00', 'runtime': '862080.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.848', 'adc_il_f': '0.321', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.8', 'ahc_t': '2654.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.781', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.8', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862084407 ms)
2025-02-17 19:23:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:22:25.400181+00:00', 'runtime': '862020.001', 'adc_vb_f': '14.508', 'adc_va_f': '21.866', 'adc_vl_f': '14.493', 'adc_ic_f': '1.846', 'adc_il_f': '0.333', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.7', 'ahc_t': '2654.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.963', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.7', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862084408 ms)
2025-02-17 19:24:24.612 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+862142431 ms)
2025-02-17 19:24:24.612 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+862142432 ms)
2025-02-17 19:24:24.613 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55904)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+862142432 ms)
2025-02-17 19:24:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862144407 ms)
2025-02-17 19:24:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862144408 ms)
2025-02-17 19:24:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:24:25.400210+00:00', 'runtime': '862140.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.87', 'adc_il_f': '0.331', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.8', 'ahc_t': '2654.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.528', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.8', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862144409 ms)
2025-02-17 19:24:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:23:25.400206+00:00', 'runtime': '862080.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.848', 'adc_il_f': '0.321', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.8', 'ahc_t': '2654.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.781', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.8', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862144409 ms)
2025-02-17 19:25:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862204407 ms)
2025-02-17 19:25:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862204407 ms)
2025-02-17 19:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:25:25.400166+00:00', 'runtime': '862200.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.875', 'adc_vl_f': '14.493', 'adc_ic_f': '1.817', 'adc_il_f': '0.304', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.8', 'ahc_t': '2654.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.74', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.8', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862204408 ms)
2025-02-17 19:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:24:25.400210+00:00', 'runtime': '862140.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.878', 'adc_vl_f': '14.493', 'adc_ic_f': '1.87', 'adc_il_f': '0.331', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.8', 'ahc_t': '2654.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.528', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.8', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862204409 ms)
2025-02-17 19:26:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862264409 ms)
2025-02-17 19:26:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862264409 ms)
2025-02-17 19:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:26:25.402727+00:00', 'runtime': '862260.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.82', 'adc_vl_f': '14.493', 'adc_ic_f': '1.838', 'adc_il_f': '0.326', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.9', 'ahc_t': '2654.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.298', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.9', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862264410 ms)
2025-02-17 19:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:25:25.400166+00:00', 'runtime': '862200.001', 'adc_vb_f': '14.511', 'adc_va_f': '21.875', 'adc_vl_f': '14.493', 'adc_ic_f': '1.817', 'adc_il_f': '0.304', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.8', 'ahc_t': '2654.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.74', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.8', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862264410 ms)
2025-02-17 19:26:35.622 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862273442 ms)
2025-02-17 19:26:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862273442 ms)
2025-02-17 19:26:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:26:25.400133+00:00'} (T+862273443 ms)
2025-02-17 19:27:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862324405 ms)
2025-02-17 19:27:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862324406 ms)
2025-02-17 19:27:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:27:25.402740+00:00', 'runtime': '862320.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.771', 'adc_vl_f': '14.453', 'adc_ic_f': '1.814', 'adc_il_f': '0.324', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.505', 'vb_ref': '14.001', 'ahc_r': '2654.9', 'ahc_t': '2654.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.449', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.9', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862324407 ms)
2025-02-17 19:27:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:26:25.402727+00:00', 'runtime': '862260.003', 'adc_vb_f': '14.511', 'adc_va_f': '21.82', 'adc_vl_f': '14.493', 'adc_ic_f': '1.838', 'adc_il_f': '0.326', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.511', 'vb_ref': '14.03', 'ahc_r': '2654.9', 'ahc_t': '2654.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.298', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.9', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862324409 ms)
2025-02-17 19:27:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862333444 ms)
2025-02-17 19:27:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862333445 ms)
2025-02-17 19:27:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:27:25.400165+00:00'} (T+862333446 ms)
2025-02-17 19:28:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862384403 ms)
2025-02-17 19:28:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862384404 ms)
2025-02-17 19:28:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:28:25.400197+00:00', 'runtime': '862380.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.771', 'adc_vl_f': '14.456', 'adc_ic_f': '1.831', 'adc_il_f': '0.336', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2654.9', 'ahc_t': '2654.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.679', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.9', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862384406 ms)
2025-02-17 19:28:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:27:25.402740+00:00', 'runtime': '862320.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.771', 'adc_vl_f': '14.453', 'adc_ic_f': '1.814', 'adc_il_f': '0.324', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.505', 'vb_ref': '14.001', 'ahc_r': '2654.9', 'ahc_t': '2654.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.449', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.9', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862384406 ms)
2025-02-17 19:28:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862393422 ms)
2025-02-17 19:28:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862393422 ms)
2025-02-17 19:28:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:28:25.400182+00:00'} (T+862393423 ms)
2025-02-17 19:29:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862444405 ms)
2025-02-17 19:29:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862444406 ms)
2025-02-17 19:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:29:25.400192+00:00', 'runtime': '862440.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.719', 'adc_vl_f': '14.456', 'adc_ic_f': '1.867', 'adc_il_f': '0.365', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.147', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862444410 ms)
2025-02-17 19:29:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:28:25.400197+00:00', 'runtime': '862380.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.771', 'adc_vl_f': '14.456', 'adc_ic_f': '1.831', 'adc_il_f': '0.336', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2654.9', 'ahc_t': '2654.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.679', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '24.9', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862444410 ms)
2025-02-17 19:29:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862453437 ms)
2025-02-17 19:29:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862453438 ms)
2025-02-17 19:29:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:29:25.400217+00:00'} (T+862453439 ms)
2025-02-17 19:30:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862504406 ms)
2025-02-17 19:30:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862504406 ms)
2025-02-17 19:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:30:25.400208+00:00', 'runtime': '862500.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.762', 'adc_vl_f': '14.456', 'adc_ic_f': '1.85', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.966', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862504408 ms)
2025-02-17 19:30:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:29:25.400192+00:00', 'runtime': '862440.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.719', 'adc_vl_f': '14.456', 'adc_ic_f': '1.867', 'adc_il_f': '0.365', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.147', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862504409 ms)
2025-02-17 19:30:35.809 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862513628 ms)
2025-02-17 19:30:35.809 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862513629 ms)
2025-02-17 19:30:35.810 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:30:25.400225+00:00'} (T+862513630 ms)
2025-02-17 19:31:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862564404 ms)
2025-02-17 19:31:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862564404 ms)
2025-02-17 19:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:31:25.400195+00:00', 'runtime': '862560.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.765', 'adc_vl_f': '14.459', 'adc_ic_f': '1.858', 'adc_il_f': '0.355', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.724', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862564406 ms)
2025-02-17 19:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:30:25.400208+00:00', 'runtime': '862500.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.762', 'adc_vl_f': '14.456', 'adc_ic_f': '1.85', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.966', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862564407 ms)
2025-02-17 19:31:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862573430 ms)
2025-02-17 19:31:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862573431 ms)
2025-02-17 19:31:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:31:25.400053+00:00'} (T+862573432 ms)
2025-02-17 19:32:25.300 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+862623119 ms)
2025-02-17 19:32:25.300 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+862623119 ms)
2025-02-17 19:32:25.301 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56424)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+862623120 ms)
2025-02-17 19:32:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862624405 ms)
2025-02-17 19:32:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862624406 ms)
2025-02-17 19:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:32:25.400246+00:00', 'runtime': '862620.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.786', 'adc_vl_f': '14.453', 'adc_ic_f': '1.836', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.072', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862624408 ms)
2025-02-17 19:32:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:31:25.400195+00:00', 'runtime': '862560.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.765', 'adc_vl_f': '14.459', 'adc_ic_f': '1.858', 'adc_il_f': '0.355', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.724', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862624408 ms)
2025-02-17 19:32:35.688 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862633507 ms)
2025-02-17 19:32:35.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862633508 ms)
2025-02-17 19:32:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:32:25.400245+00:00'} (T+862633509 ms)
2025-02-17 19:33:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862684421 ms)
2025-02-17 19:33:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862684421 ms)
2025-02-17 19:33:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:33:25.402714+00:00', 'runtime': '862680.003', 'adc_vb_f': '14.481', 'adc_va_f': '21.878', 'adc_vl_f': '14.453', 'adc_ic_f': '1.797', 'adc_il_f': '0.336', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.1', 'ahc_t': '2655.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.906', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.1', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862684425 ms)
2025-02-17 19:33:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:32:25.400246+00:00', 'runtime': '862620.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.786', 'adc_vl_f': '14.453', 'adc_ic_f': '1.836', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.0', 'ahc_t': '2655.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.072', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.0', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862684426 ms)
2025-02-17 19:33:35.874 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862693694 ms)
2025-02-17 19:33:35.875 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862693694 ms)
2025-02-17 19:33:35.877 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:33:25.400131+00:00'} (T+862693696 ms)
2025-02-17 19:33:55.416 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+862713235 ms)
2025-02-17 19:33:55.416 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+862713235 ms)
2025-02-17 19:33:55.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+862713236 ms)
2025-02-17 19:33:55.418 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+862713238 ms)
2025-02-17 19:33:55.419 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+862713238 ms)
2025-02-17 19:33:55.419 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+862713238 ms)
2025-02-17 19:34:25.500 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+862743319 ms)
2025-02-17 19:34:25.500 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+862743319 ms)
2025-02-17 19:34:25.501 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56554)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+862743320 ms)
2025-02-17 19:34:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862744418 ms)
2025-02-17 19:34:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862744418 ms)
2025-02-17 19:34:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:34:25.402824+00:00', 'runtime': '862740.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.878', 'adc_vl_f': '14.453', 'adc_ic_f': '1.785', 'adc_il_f': '0.321', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.1', 'ahc_t': '2655.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.649', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.1', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862744422 ms)
2025-02-17 19:34:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:33:25.402714+00:00', 'runtime': '862680.003', 'adc_vb_f': '14.481', 'adc_va_f': '21.878', 'adc_vl_f': '14.453', 'adc_ic_f': '1.797', 'adc_il_f': '0.336', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.1', 'ahc_t': '2655.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.906', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.1', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862744422 ms)
2025-02-17 19:34:35.740 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862753560 ms)
2025-02-17 19:34:35.741 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862753560 ms)
2025-02-17 19:34:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:34:25.400199+00:00'} (T+862753561 ms)
2025-02-17 19:35:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862804419 ms)
2025-02-17 19:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862804420 ms)
2025-02-17 19:35:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:35:25.402686+00:00', 'runtime': '862800.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.912', 'adc_vl_f': '14.453', 'adc_ic_f': '1.785', 'adc_il_f': '0.316', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2655.1', 'ahc_t': '2655.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.332', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.1', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862804424 ms)
2025-02-17 19:35:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:34:25.402824+00:00', 'runtime': '862740.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.878', 'adc_vl_f': '14.453', 'adc_ic_f': '1.785', 'adc_il_f': '0.321', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.1', 'ahc_t': '2655.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.649', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.1', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862804424 ms)
2025-02-17 19:35:35.937 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862813756 ms)
2025-02-17 19:35:35.937 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862813757 ms)
2025-02-17 19:35:35.938 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:35:25.400069+00:00'} (T+862813757 ms)
2025-02-17 19:36:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862864421 ms)
2025-02-17 19:36:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862864421 ms)
2025-02-17 19:36:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:36:25.402749+00:00', 'runtime': '862860.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.899', 'adc_vl_f': '14.453', 'adc_ic_f': '1.773', 'adc_il_f': '0.326', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.2', 'ahc_t': '2655.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.377', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.2', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862864423 ms)
2025-02-17 19:36:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:35:25.402686+00:00', 'runtime': '862800.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.912', 'adc_vl_f': '14.453', 'adc_ic_f': '1.785', 'adc_il_f': '0.316', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2655.1', 'ahc_t': '2655.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.332', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.1', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862864423 ms)
2025-02-17 19:36:35.925 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862873744 ms)
2025-02-17 19:36:35.927 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862873747 ms)
2025-02-17 19:36:35.928 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:36:25.400185+00:00'} (T+862873748 ms)
2025-02-17 19:37:05.756 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+862903576 ms)
2025-02-17 19:37:05.757 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+862903576 ms)
2025-02-17 19:37:05.757 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56738)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+862903577 ms)
2025-02-17 19:37:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862924418 ms)
2025-02-17 19:37:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862924419 ms)
2025-02-17 19:37:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:37:25.402585+00:00', 'runtime': '862920.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.899', 'adc_vl_f': '14.453', 'adc_ic_f': '1.821', 'adc_il_f': '0.365', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.2', 'ahc_t': '2655.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.377', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.2', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862924421 ms)
2025-02-17 19:37:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:36:25.402749+00:00', 'runtime': '862860.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.899', 'adc_vl_f': '14.453', 'adc_ic_f': '1.773', 'adc_il_f': '0.326', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.2', 'ahc_t': '2655.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.7', 'ahl_t': '1591.7', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.377', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.2', 'ahl_daily': '7.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862924421 ms)
2025-02-17 19:37:35.900 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862933719 ms)
2025-02-17 19:37:35.901 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862933720 ms)
2025-02-17 19:37:35.902 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:37:25.399961+00:00'} (T+862933721 ms)
2025-02-17 19:38:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+862984420 ms)
2025-02-17 19:38:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+862984420 ms)
2025-02-17 19:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:38:25.401160+00:00', 'runtime': '862980.002', 'adc_vb_f': '14.484', 'adc_va_f': '21.957', 'adc_vl_f': '14.453', 'adc_ic_f': '1.783', 'adc_il_f': '0.331', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.2', 'ahc_t': '2655.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.332', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.2', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862984422 ms)
2025-02-17 19:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:37:25.402585+00:00', 'runtime': '862920.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.899', 'adc_vl_f': '14.453', 'adc_ic_f': '1.821', 'adc_il_f': '0.365', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.2', 'ahc_t': '2655.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.377', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.2', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+862984423 ms)
2025-02-17 19:38:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862993431 ms)
2025-02-17 19:38:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+862993432 ms)
2025-02-17 19:38:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:38:25.400185+00:00'} (T+862993434 ms)
2025-02-17 19:39:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863044420 ms)
2025-02-17 19:39:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863044420 ms)
2025-02-17 19:39:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:39:25.400173+00:00', 'runtime': '863040.001', 'adc_vb_f': '14.487', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.802', 'adc_il_f': '0.355', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.985', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863044423 ms)
2025-02-17 19:39:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:38:25.401160+00:00', 'runtime': '862980.002', 'adc_vb_f': '14.484', 'adc_va_f': '21.957', 'adc_vl_f': '14.453', 'adc_ic_f': '1.783', 'adc_il_f': '0.331', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.2', 'ahc_t': '2655.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.332', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.2', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863044423 ms)
2025-02-17 19:39:35.727 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863053547 ms)
2025-02-17 19:39:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863053547 ms)
2025-02-17 19:39:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:39:25.413123+00:00'} (T+863053548 ms)
2025-02-17 19:40:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863104421 ms)
2025-02-17 19:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863104421 ms)
2025-02-17 19:40:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:40:25.402672+00:00', 'runtime': '863100.003', 'adc_vb_f': '14.481', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.814', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.558', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863104423 ms)
2025-02-17 19:40:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:39:25.400173+00:00', 'runtime': '863040.001', 'adc_vb_f': '14.487', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.802', 'adc_il_f': '0.355', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.985', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863104424 ms)
2025-02-17 19:40:58.048 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+863135867 ms)
2025-02-17 19:40:58.048 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+863135867 ms)
2025-02-17 19:40:58.048 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863135868 ms)
2025-02-17 19:40:58.049 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+863135868 ms)
2025-02-17 19:40:58.049 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+863135869 ms)
2025-02-17 19:40:58.050 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863135869 ms)
2025-02-17 19:41:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863164403 ms)
2025-02-17 19:41:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863164404 ms)
2025-02-17 19:41:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:41:25.400594+00:00', 'runtime': '863160.001', 'adc_vb_f': '14.487', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.788', 'adc_il_f': '0.345', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.015', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863164407 ms)
2025-02-17 19:41:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:40:25.402672+00:00', 'runtime': '863100.003', 'adc_vb_f': '14.481', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.814', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.558', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863164407 ms)
2025-02-17 19:41:58.120 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+863195939 ms)
2025-02-17 19:41:58.120 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+863195939 ms)
2025-02-17 19:41:58.121 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863195940 ms)
2025-02-17 19:42:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863224418 ms)
2025-02-17 19:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863224418 ms)
2025-02-17 19:42:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:42:25.402839+00:00', 'runtime': '863220.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.902', 'adc_vl_f': '14.453', 'adc_ic_f': '1.807', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.196', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863224422 ms)
2025-02-17 19:42:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:41:25.400594+00:00', 'runtime': '863160.001', 'adc_vb_f': '14.487', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.788', 'adc_il_f': '0.345', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.015', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863224423 ms)
2025-02-17 19:43:13.326 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863271146 ms)
2025-02-17 19:43:13.327 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863271146 ms)
2025-02-17 19:43:13.327 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57128)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863271146 ms)
2025-02-17 19:43:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863284401 ms)
2025-02-17 19:43:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863284401 ms)
2025-02-17 19:43:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:43:25.400163+00:00', 'runtime': '863280.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.951', 'adc_vl_f': '14.453', 'adc_ic_f': '1.764', 'adc_il_f': '0.341', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.4', 'ahc_t': '2655.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.939', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.4', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863284402 ms)
2025-02-17 19:43:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:42:25.402839+00:00', 'runtime': '863220.003', 'adc_vb_f': '14.484', 'adc_va_f': '21.902', 'adc_vl_f': '14.453', 'adc_ic_f': '1.807', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.3', 'ahc_t': '2655.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.196', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.3', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863284403 ms)
2025-02-17 19:44:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863344417 ms)
2025-02-17 19:44:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863344418 ms)
2025-02-17 19:44:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:44:25.400204+00:00', 'runtime': '863340.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.754', 'adc_il_f': '0.326', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.4', 'ahc_t': '2655.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.849', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.4', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863344419 ms)
2025-02-17 19:44:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:43:25.400163+00:00', 'runtime': '863280.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.951', 'adc_vl_f': '14.453', 'adc_ic_f': '1.764', 'adc_il_f': '0.341', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.4', 'ahc_t': '2655.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.939', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.4', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863344419 ms)
2025-02-17 19:44:35.736 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863353556 ms)
2025-02-17 19:44:35.737 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863353556 ms)
2025-02-17 19:44:35.738 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:44:25.400338+00:00'} (T+863353557 ms)
2025-02-17 19:45:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863404416 ms)
2025-02-17 19:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863404416 ms)
2025-02-17 19:45:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:45:25.400177+00:00', 'runtime': '863400.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.732', 'adc_il_f': '0.316', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2655.4', 'ahc_t': '2655.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.909', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.4', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863404417 ms)
2025-02-17 19:45:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:44:25.400204+00:00', 'runtime': '863340.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.754', 'adc_il_f': '0.326', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.4', 'ahc_t': '2655.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.849', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.4', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863404418 ms)
2025-02-17 19:45:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863413430 ms)
2025-02-17 19:45:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863413430 ms)
2025-02-17 19:45:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:45:25.400074+00:00'} (T+863413431 ms)
2025-02-17 19:46:18.649 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863456468 ms)
2025-02-17 19:46:18.649 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863456469 ms)
2025-02-17 19:46:18.650 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57334)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863456469 ms)
2025-02-17 19:46:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863464417 ms)
2025-02-17 19:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863464418 ms)
2025-02-17 19:46:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:46:25.400164+00:00', 'runtime': '863460.001', 'adc_vb_f': '14.481', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.73', 'adc_il_f': '0.341', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.5', 'ahc_t': '2655.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.06', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.5', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863464419 ms)
2025-02-17 19:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:45:25.400177+00:00', 'runtime': '863400.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.732', 'adc_il_f': '0.316', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2655.4', 'ahc_t': '2655.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.909', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.4', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863464419 ms)
2025-02-17 19:46:56.797 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863494616 ms)
2025-02-17 19:46:56.797 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863494616 ms)
2025-02-17 19:46:56.798 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57378)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863494617 ms)
2025-02-17 19:47:26.581 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863524401 ms)
2025-02-17 19:47:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863524401 ms)
2025-02-17 19:47:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:47:25.400160+00:00', 'runtime': '863520.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.727', 'adc_il_f': '0.309', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2655.5', 'ahc_t': '2655.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.701', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.5', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863524402 ms)
2025-02-17 19:47:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:46:25.400164+00:00', 'runtime': '863460.001', 'adc_vb_f': '14.481', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.73', 'adc_il_f': '0.341', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.484', 'vb_ref': '14.001', 'ahc_r': '2655.5', 'ahc_t': '2655.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.06', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.5', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863524403 ms)
2025-02-17 19:47:33.908 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863531728 ms)
2025-02-17 19:47:33.909 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863531728 ms)
2025-02-17 19:47:33.909 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57416)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863531728 ms)
2025-02-17 19:48:19.019 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863576838 ms)
2025-02-17 19:48:19.019 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863576838 ms)
2025-02-17 19:48:19.019 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57464)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863576839 ms)
2025-02-17 19:48:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863584402 ms)
2025-02-17 19:48:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863584403 ms)
2025-02-17 19:48:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:48:25.400175+00:00', 'runtime': '863580.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.771', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.471', 'vb_ref': '13.971', 'ahc_r': '2655.5', 'ahc_t': '2655.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.607', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.5', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863584403 ms)
2025-02-17 19:48:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:47:25.400160+00:00', 'runtime': '863520.001', 'adc_vb_f': '14.484', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.727', 'adc_il_f': '0.309', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.487', 'vb_ref': '14.001', 'ahc_r': '2655.5', 'ahc_t': '2655.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.701', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.5', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863584404 ms)
2025-02-17 19:48:36.352 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863594171 ms)
2025-02-17 19:48:36.352 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863594172 ms)
2025-02-17 19:48:36.353 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:48:25.400349+00:00'} (T+863594173 ms)
2025-02-17 19:49:19.167 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863636986 ms)
2025-02-17 19:49:19.167 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863636986 ms)
2025-02-17 19:49:19.167 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57530)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863636987 ms)
2025-02-17 19:49:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863644418 ms)
2025-02-17 19:49:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863644418 ms)
2025-02-17 19:49:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:49:25.400188+00:00', 'runtime': '863640.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.964', 'adc_vl_f': '14.453', 'adc_ic_f': '1.764', 'adc_il_f': '0.36', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.462', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.426', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863644419 ms)
2025-02-17 19:49:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:48:25.400175+00:00', 'runtime': '863580.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.771', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.471', 'vb_ref': '13.971', 'ahc_r': '2655.5', 'ahc_t': '2655.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.607', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.5', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863644419 ms)
2025-02-17 19:49:36.147 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863653966 ms)
2025-02-17 19:49:36.147 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863653967 ms)
2025-02-17 19:49:36.148 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:49:25.400471+00:00'} (T+863653967 ms)
2025-02-17 19:50:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863704403 ms)
2025-02-17 19:50:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863704404 ms)
2025-02-17 19:50:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:50:25.400189+00:00', 'runtime': '863700.001', 'adc_vb_f': '14.447', 'adc_va_f': '21.982', 'adc_vl_f': '14.426', 'adc_ic_f': '1.544', 'adc_il_f': '0.358', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.535', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863704405 ms)
2025-02-17 19:50:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:49:25.400188+00:00', 'runtime': '863640.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.964', 'adc_vl_f': '14.453', 'adc_ic_f': '1.764', 'adc_il_f': '0.36', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.462', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.426', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863704405 ms)
2025-02-17 19:50:35.745 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863713564 ms)
2025-02-17 19:50:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863713564 ms)
2025-02-17 19:50:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:50:25.400189+00:00'} (T+863713565 ms)
2025-02-17 19:51:00.379 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863738198 ms)
2025-02-17 19:51:00.379 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863738198 ms)
2025-02-17 19:51:00.379 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57640)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863738198 ms)
2025-02-17 19:51:16.485 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863754304 ms)
2025-02-17 19:51:16.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863754305 ms)
2025-02-17 19:51:16.486 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57658)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863754305 ms)
2025-02-17 19:51:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863764405 ms)
2025-02-17 19:51:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863764405 ms)
2025-02-17 19:51:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:51:25.400185+00:00', 'runtime': '863760.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.921', 'adc_vl_f': '14.453', 'adc_ic_f': '1.679', 'adc_il_f': '0.314', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.353', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863764406 ms)
2025-02-17 19:51:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:50:25.400189+00:00', 'runtime': '863700.001', 'adc_vb_f': '14.447', 'adc_va_f': '21.982', 'adc_vl_f': '14.426', 'adc_ic_f': '1.544', 'adc_il_f': '0.358', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.535', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863764407 ms)
2025-02-17 19:51:36.503 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863774322 ms)
2025-02-17 19:51:36.504 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863774323 ms)
2025-02-17 19:51:36.505 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:51:25.400052+00:00'} (T+863774324 ms)
2025-02-17 19:52:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863824404 ms)
2025-02-17 19:52:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863824405 ms)
2025-02-17 19:52:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:52:25.400168+00:00', 'runtime': '863820.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.878', 'adc_vl_f': '14.453', 'adc_ic_f': '1.751', 'adc_il_f': '0.35', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.462', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.29', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863824406 ms)
2025-02-17 19:52:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:51:25.400185+00:00', 'runtime': '863760.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.921', 'adc_vl_f': '14.453', 'adc_ic_f': '1.679', 'adc_il_f': '0.314', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.353', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863824406 ms)
2025-02-17 19:52:36.029 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863833848 ms)
2025-02-17 19:52:36.029 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863833849 ms)
2025-02-17 19:52:36.030 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:52:25.400294+00:00'} (T+863833849 ms)
2025-02-17 19:52:49.666 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863847485 ms)
2025-02-17 19:52:49.666 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863847485 ms)
2025-02-17 19:52:49.666 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57758)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863847485 ms)
2025-02-17 19:53:25.757 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+863883577 ms)
2025-02-17 19:53:25.758 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+863883577 ms)
2025-02-17 19:53:25.758 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57798)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+863883577 ms)
2025-02-17 19:53:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863884408 ms)
2025-02-17 19:53:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863884408 ms)
2025-02-17 19:53:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:53:25.400173+00:00', 'runtime': '863880.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.783', 'adc_vl_f': '14.447', 'adc_ic_f': '1.732', 'adc_il_f': '0.324', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.7', 'ahc_t': '2655.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.033', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.7', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863884409 ms)
2025-02-17 19:53:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:52:25.400168+00:00', 'runtime': '863820.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.878', 'adc_vl_f': '14.453', 'adc_ic_f': '1.751', 'adc_il_f': '0.35', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.462', 'vb_ref': '13.971', 'ahc_r': '2655.6', 'ahc_t': '2655.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.29', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.6', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863884410 ms)
2025-02-17 19:53:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863893436 ms)
2025-02-17 19:53:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863893436 ms)
2025-02-17 19:53:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:53:25.400274+00:00'} (T+863893438 ms)
2025-02-17 19:54:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+863944405 ms)
2025-02-17 19:54:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+863944406 ms)
2025-02-17 19:54:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:54:25.400170+00:00', 'runtime': '863940.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.768', 'adc_vl_f': '14.453', 'adc_ic_f': '1.725', 'adc_il_f': '0.338', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.7', 'ahc_t': '2655.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.898', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.7', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863944407 ms)
2025-02-17 19:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:53:25.400173+00:00', 'runtime': '863880.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.783', 'adc_vl_f': '14.447', 'adc_ic_f': '1.732', 'adc_il_f': '0.324', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.7', 'ahc_t': '2655.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20126', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.033', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.7', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+863944407 ms)
2025-02-17 19:54:35.816 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863953635 ms)
2025-02-17 19:54:35.817 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+863953636 ms)
2025-02-17 19:54:35.818 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:54:25.400182+00:00'} (T+863953637 ms)
2025-02-17 19:55:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864004407 ms)
2025-02-17 19:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864004407 ms)
2025-02-17 19:55:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:55:25.400206+00:00', 'runtime': '864000.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.689', 'adc_vl_f': '14.453', 'adc_ic_f': '1.742', 'adc_il_f': '0.341', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.7', 'ahc_t': '2655.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.184', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.7', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864004408 ms)
2025-02-17 19:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:54:25.400170+00:00', 'runtime': '863940.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.768', 'adc_vl_f': '14.453', 'adc_ic_f': '1.725', 'adc_il_f': '0.338', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.7', 'ahc_t': '2655.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.898', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.7', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864004409 ms)
2025-02-17 19:55:48.997 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+864026817 ms)
2025-02-17 19:55:48.998 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+864026817 ms)
2025-02-17 19:55:48.998 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57960)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+864026818 ms)
2025-02-17 19:56:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864064409 ms)
2025-02-17 19:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864064409 ms)
2025-02-17 19:56:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:56:25.400172+00:00', 'runtime': '864060.001', 'adc_vb_f': '14.45', 'adc_va_f': '21.762', 'adc_vl_f': '14.42', 'adc_ic_f': '1.626', 'adc_il_f': '0.365', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.958', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864064410 ms)
2025-02-17 19:56:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:55:25.400206+00:00', 'runtime': '864000.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.689', 'adc_vl_f': '14.453', 'adc_ic_f': '1.742', 'adc_il_f': '0.341', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.7', 'ahc_t': '2655.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.184', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.7', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864064411 ms)
2025-02-17 19:57:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864124410 ms)
2025-02-17 19:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864124410 ms)
2025-02-17 19:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:57:25.400171+00:00', 'runtime': '864120.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.875', 'adc_vl_f': '14.453', 'adc_ic_f': '1.735', 'adc_il_f': '0.353', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.079', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864124411 ms)
2025-02-17 19:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:56:25.400172+00:00', 'runtime': '864060.001', 'adc_vb_f': '14.45', 'adc_va_f': '21.762', 'adc_vl_f': '14.42', 'adc_ic_f': '1.626', 'adc_il_f': '0.365', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.8', 'ahl_t': '1591.8', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.958', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864124412 ms)
2025-02-17 19:57:35.665 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864133484 ms)
2025-02-17 19:57:35.665 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864133485 ms)
2025-02-17 19:57:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:57:25.400336+00:00'} (T+864133486 ms)
2025-02-17 19:57:52.205 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+864150025 ms)
2025-02-17 19:57:52.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+864150025 ms)
2025-02-17 19:57:52.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58094)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+864150025 ms)
2025-02-17 19:58:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864184411 ms)
2025-02-17 19:58:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864184412 ms)
2025-02-17 19:58:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:58:25.400204+00:00', 'runtime': '864180.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.896', 'adc_vl_f': '14.453', 'adc_ic_f': '1.718', 'adc_il_f': '0.355', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.837', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864184413 ms)
2025-02-17 19:58:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:57:25.400171+00:00', 'runtime': '864120.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.875', 'adc_vl_f': '14.453', 'adc_ic_f': '1.735', 'adc_il_f': '0.353', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.079', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864184414 ms)
2025-02-17 19:58:35.733 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864193553 ms)
2025-02-17 19:58:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864193553 ms)
2025-02-17 19:58:35.735 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:58:25.400182+00:00'} (T+864193554 ms)
2025-02-17 19:59:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864244411 ms)
2025-02-17 19:59:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864244411 ms)
2025-02-17 19:59:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 19:59:25.400173+00:00', 'runtime': '864240.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.722', 'adc_il_f': '0.343', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.867', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864244412 ms)
2025-02-17 19:59:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:58:25.400204+00:00', 'runtime': '864180.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.896', 'adc_vl_f': '14.453', 'adc_ic_f': '1.718', 'adc_il_f': '0.355', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.837', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864244413 ms)
2025-02-17 19:59:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864253418 ms)
2025-02-17 19:59:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864253419 ms)
2025-02-17 19:59:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 19:59:25.400069+00:00'} (T+864253420 ms)
2025-02-17 20:00:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864304411 ms)
2025-02-17 20:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864304411 ms)
2025-02-17 20:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:00:25.400181+00:00', 'runtime': '864300.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.997', 'adc_vl_f': '14.453', 'adc_ic_f': '1.725', 'adc_il_f': '0.326', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.9', 'ahc_t': '2655.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.898', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.9', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864304412 ms)
2025-02-17 20:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 19:59:25.400173+00:00', 'runtime': '864240.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.988', 'adc_vl_f': '14.453', 'adc_ic_f': '1.722', 'adc_il_f': '0.343', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.8', 'ahc_t': '2655.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.867', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.8', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864304413 ms)
2025-02-17 20:00:35.415 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864313234 ms)
2025-02-17 20:00:35.415 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864313235 ms)
2025-02-17 20:00:35.416 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:00:25.400214+00:00'} (T+864313236 ms)
2025-02-17 20:01:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864364412 ms)
2025-02-17 20:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864364412 ms)
2025-02-17 20:01:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:01:25.400162+00:00', 'runtime': '864360.001', 'adc_vb_f': '14.453', 'adc_va_f': '22.083', 'adc_vl_f': '14.435', 'adc_ic_f': '1.437', 'adc_il_f': '0.312', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.9', 'ahc_t': '2655.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.074', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.9', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864364413 ms)
2025-02-17 20:01:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:00:25.400181+00:00', 'runtime': '864300.001', 'adc_vb_f': '14.462', 'adc_va_f': '21.997', 'adc_vl_f': '14.453', 'adc_ic_f': '1.725', 'adc_il_f': '0.326', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.9', 'ahc_t': '2655.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.898', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.9', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864364414 ms)
2025-02-17 20:01:35.455 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864373274 ms)
2025-02-17 20:01:35.455 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864373274 ms)
2025-02-17 20:01:35.456 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:01:25.400401+00:00'} (T+864373276 ms)
2025-02-17 20:01:37.565 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+864375385 ms)
2025-02-17 20:01:37.566 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+864375385 ms)
2025-02-17 20:01:37.566 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58334)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+864375385 ms)
2025-02-17 20:02:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864424413 ms)
2025-02-17 20:02:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864424414 ms)
2025-02-17 20:02:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:02:25.400159+00:00', 'runtime': '864420.001', 'adc_vb_f': '14.459', 'adc_va_f': '22.067', 'adc_vl_f': '14.441', 'adc_ic_f': '1.52', 'adc_il_f': '0.321', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.9', 'ahc_t': '2655.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.9', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864424415 ms)
2025-02-17 20:02:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:01:25.400162+00:00', 'runtime': '864360.001', 'adc_vb_f': '14.453', 'adc_va_f': '22.083', 'adc_vl_f': '14.435', 'adc_ic_f': '1.437', 'adc_il_f': '0.312', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.9', 'ahc_t': '2655.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.074', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.9', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864424415 ms)
2025-02-17 20:02:35.437 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864433256 ms)
2025-02-17 20:02:35.438 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864433257 ms)
2025-02-17 20:02:35.439 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:02:25.400353+00:00'} (T+864433258 ms)
2025-02-17 20:03:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864484413 ms)
2025-02-17 20:03:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864484413 ms)
2025-02-17 20:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:03:25.400159+00:00', 'runtime': '864480.001', 'adc_vb_f': '14.459', 'adc_va_f': '22.079', 'adc_vl_f': '14.444', 'adc_ic_f': '1.689', 'adc_il_f': '0.321', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.52', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864484414 ms)
2025-02-17 20:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:02:25.400159+00:00', 'runtime': '864420.001', 'adc_vb_f': '14.459', 'adc_va_f': '22.067', 'adc_vl_f': '14.441', 'adc_ic_f': '1.52', 'adc_il_f': '0.321', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2655.9', 'ahc_t': '2655.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '25.9', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864484415 ms)
2025-02-17 20:04:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864544414 ms)
2025-02-17 20:04:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864544414 ms)
2025-02-17 20:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:04:25.400172+00:00', 'runtime': '864540.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.009', 'adc_vl_f': '14.453', 'adc_ic_f': '1.701', 'adc_il_f': '0.326', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.596', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864544415 ms)
2025-02-17 20:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:03:25.400159+00:00', 'runtime': '864480.001', 'adc_vb_f': '14.459', 'adc_va_f': '22.079', 'adc_vl_f': '14.444', 'adc_ic_f': '1.689', 'adc_il_f': '0.321', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.52', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864544416 ms)
2025-02-17 20:05:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864604416 ms)
2025-02-17 20:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864604416 ms)
2025-02-17 20:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:05:25.400225+00:00', 'runtime': '864600.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.012', 'adc_vl_f': '14.453', 'adc_ic_f': '1.715', 'adc_il_f': '0.333', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.837', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864604417 ms)
2025-02-17 20:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:04:25.400172+00:00', 'runtime': '864540.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.009', 'adc_vl_f': '14.453', 'adc_ic_f': '1.701', 'adc_il_f': '0.326', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.596', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864604418 ms)
2025-02-17 20:06:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864664420 ms)
2025-02-17 20:06:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864664420 ms)
2025-02-17 20:06:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:06:25.400182+00:00', 'runtime': '864660.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.092', 'adc_vl_f': '14.453', 'adc_ic_f': '1.708', 'adc_il_f': '0.37', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.731', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864664423 ms)
2025-02-17 20:06:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:05:25.400225+00:00', 'runtime': '864600.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.012', 'adc_vl_f': '14.453', 'adc_ic_f': '1.715', 'adc_il_f': '0.333', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.837', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864664424 ms)
2025-02-17 20:07:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864724405 ms)
2025-02-17 20:07:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864724406 ms)
2025-02-17 20:07:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:07:25.400160+00:00', 'runtime': '864720.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.058', 'adc_vl_f': '14.453', 'adc_ic_f': '1.706', 'adc_il_f': '0.36', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.1', 'ahc_t': '2656.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.656', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.1', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864724410 ms)
2025-02-17 20:07:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:06:25.400182+00:00', 'runtime': '864660.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.092', 'adc_vl_f': '14.453', 'adc_ic_f': '1.708', 'adc_il_f': '0.37', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.0', 'ahc_t': '2656.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.731', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.0', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864724411 ms)
2025-02-17 20:08:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864784407 ms)
2025-02-17 20:08:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864784407 ms)
2025-02-17 20:08:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:08:25.400577+00:00', 'runtime': '864780.001', 'adc_vb_f': '14.417', 'adc_va_f': '22.089', 'adc_vl_f': '14.38', 'adc_ic_f': '1.546', 'adc_il_f': '0.35', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.45', 'vb_ref': '13.942', 'ahc_r': '2656.1', 'ahc_t': '2656.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.037', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.1', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864784411 ms)
2025-02-17 20:08:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:07:25.400160+00:00', 'runtime': '864720.001', 'adc_vb_f': '14.462', 'adc_va_f': '22.058', 'adc_vl_f': '14.453', 'adc_ic_f': '1.706', 'adc_il_f': '0.36', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.459', 'vb_ref': '13.971', 'ahc_r': '2656.1', 'ahc_t': '2656.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.656', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.1', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864784412 ms)
2025-02-17 20:09:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864844422 ms)
2025-02-17 20:09:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864844423 ms)
2025-02-17 20:09:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:09:25.402879+00:00', 'runtime': '864840.003', 'adc_vb_f': '14.42', 'adc_va_f': '21.991', 'adc_vl_f': '14.377', 'adc_ic_f': '1.674', 'adc_il_f': '0.367', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.1', 'ahc_t': '2656.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.731', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.1', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864844426 ms)
2025-02-17 20:09:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:08:25.400577+00:00', 'runtime': '864780.001', 'adc_vb_f': '14.417', 'adc_va_f': '22.089', 'adc_vl_f': '14.38', 'adc_ic_f': '1.546', 'adc_il_f': '0.35', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.45', 'vb_ref': '13.942', 'ahc_r': '2656.1', 'ahc_t': '2656.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.037', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.1', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864844426 ms)
2025-02-17 20:09:35.695 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864853514 ms)
2025-02-17 20:09:35.696 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864853515 ms)
2025-02-17 20:09:35.696 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:09:25.400191+00:00'} (T+864853516 ms)
2025-02-17 20:10:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864904422 ms)
2025-02-17 20:10:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864904423 ms)
2025-02-17 20:10:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:10:25.400091+00:00', 'runtime': '864900.001', 'adc_vb_f': '14.42', 'adc_va_f': '21.991', 'adc_vl_f': '14.377', 'adc_ic_f': '1.645', 'adc_il_f': '0.355', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.686', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864904427 ms)
2025-02-17 20:10:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:09:25.402879+00:00', 'runtime': '864840.003', 'adc_vb_f': '14.42', 'adc_va_f': '21.991', 'adc_vl_f': '14.377', 'adc_ic_f': '1.674', 'adc_il_f': '0.367', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.1', 'ahc_t': '2656.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.731', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.1', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864904427 ms)
2025-02-17 20:10:33.354 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+864911174 ms)
2025-02-17 20:10:33.355 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+864911174 ms)
2025-02-17 20:10:33.355 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58906)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+864911174 ms)
2025-02-17 20:10:35.833 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864913652 ms)
2025-02-17 20:10:35.833 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864913653 ms)
2025-02-17 20:10:35.835 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:10:25.413121+00:00'} (T+864913654 ms)
2025-02-17 20:10:51.378 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+864929197 ms)
2025-02-17 20:10:51.378 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+864929197 ms)
2025-02-17 20:10:51.378 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+864929197 ms)
2025-02-17 20:11:22.485 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+864960304 ms)
2025-02-17 20:11:22.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+864960304 ms)
2025-02-17 20:11:22.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58960)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+864960305 ms)
2025-02-17 20:11:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+864964408 ms)
2025-02-17 20:11:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+864964408 ms)
2025-02-17 20:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:11:25.402651+00:00', 'runtime': '864960.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.924', 'adc_vl_f': '14.417', 'adc_ic_f': '1.657', 'adc_il_f': '0.324', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.871', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864964411 ms)
2025-02-17 20:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:10:25.400091+00:00', 'runtime': '864900.001', 'adc_vb_f': '14.42', 'adc_va_f': '21.991', 'adc_vl_f': '14.377', 'adc_ic_f': '1.645', 'adc_il_f': '0.355', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.686', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+864964412 ms)
2025-02-17 20:11:35.887 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864973706 ms)
2025-02-17 20:11:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+864973707 ms)
2025-02-17 20:11:35.889 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:11:25.400024+00:00'} (T+864973708 ms)
2025-02-17 20:12:07.623 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+865005443 ms)
2025-02-17 20:12:07.624 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+865005443 ms)
2025-02-17 20:12:07.624 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59010)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+865005443 ms)
2025-02-17 20:12:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865024406 ms)
2025-02-17 20:12:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865024407 ms)
2025-02-17 20:12:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:12:25.402839+00:00', 'runtime': '865020.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.988', 'adc_vl_f': '14.404', 'adc_ic_f': '1.669', 'adc_il_f': '0.326', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.992', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865024410 ms)
2025-02-17 20:12:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:11:25.402651+00:00', 'runtime': '864960.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.924', 'adc_vl_f': '14.417', 'adc_ic_f': '1.657', 'adc_il_f': '0.324', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.871', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865024411 ms)
2025-02-17 20:12:35.730 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865033550 ms)
2025-02-17 20:12:35.731 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865033550 ms)
2025-02-17 20:12:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:12:25.400201+00:00'} (T+865033551 ms)
2025-02-17 20:13:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865084424 ms)
2025-02-17 20:13:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865084425 ms)
2025-02-17 20:13:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:13:25.401121+00:00', 'runtime': '865080.002', 'adc_vb_f': '14.423', 'adc_va_f': '22.095', 'adc_vl_f': '14.377', 'adc_ic_f': '1.679', 'adc_il_f': '0.333', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.203', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865084429 ms)
2025-02-17 20:13:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:12:25.402839+00:00', 'runtime': '865020.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.988', 'adc_vl_f': '14.404', 'adc_ic_f': '1.669', 'adc_il_f': '0.326', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.992', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865084429 ms)
2025-02-17 20:14:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865144412 ms)
2025-02-17 20:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865144412 ms)
2025-02-17 20:14:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:14:25.400578+00:00', 'runtime': '865140.001', 'adc_vb_f': '14.423', 'adc_va_f': '22.113', 'adc_vl_f': '14.38', 'adc_ic_f': '1.691', 'adc_il_f': '0.309', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.354', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865144416 ms)
2025-02-17 20:14:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:13:25.401121+00:00', 'runtime': '865080.002', 'adc_vb_f': '14.423', 'adc_va_f': '22.095', 'adc_vl_f': '14.377', 'adc_ic_f': '1.679', 'adc_il_f': '0.333', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.2', 'ahc_t': '2656.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.203', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.2', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865144416 ms)
2025-02-17 20:15:25.690 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865203509 ms)
2025-02-17 20:15:25.690 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865203510 ms)
2025-02-17 20:15:25.693 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:15:25.403084+00:00', 'runtime': '865200.004', 'adc_vb_f': '14.423', 'adc_va_f': '22.107', 'adc_vl_f': '14.377', 'adc_ic_f': '1.681', 'adc_il_f': '0.358', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.339', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865203512 ms)
2025-02-17 20:15:25.693 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:14:25.400578+00:00', 'runtime': '865140.001', 'adc_vb_f': '14.423', 'adc_va_f': '22.113', 'adc_vl_f': '14.38', 'adc_ic_f': '1.691', 'adc_il_f': '0.309', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.354', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865203512 ms)
2025-02-17 20:15:35.666 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865213485 ms)
2025-02-17 20:15:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865213485 ms)
2025-02-17 20:15:35.667 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:15:25.400444+00:00'} (T+865213486 ms)
2025-02-17 20:16:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865264411 ms)
2025-02-17 20:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865264411 ms)
2025-02-17 20:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:16:25.400061+00:00', 'runtime': '865260.001', 'adc_vb_f': '14.423', 'adc_va_f': '22.095', 'adc_vl_f': '14.377', 'adc_ic_f': '1.657', 'adc_il_f': '0.358', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.856', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865264413 ms)
2025-02-17 20:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:15:25.403084+00:00', 'runtime': '865200.004', 'adc_vb_f': '14.423', 'adc_va_f': '22.107', 'adc_vl_f': '14.377', 'adc_ic_f': '1.681', 'adc_il_f': '0.358', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1591.9', 'ahl_t': '1591.9', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.339', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865264414 ms)
2025-02-17 20:16:35.732 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865273551 ms)
2025-02-17 20:16:35.733 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865273552 ms)
2025-02-17 20:16:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:16:25.401150+00:00'} (T+865273553 ms)
2025-02-17 20:17:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865324414 ms)
2025-02-17 20:17:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865324414 ms)
2025-02-17 20:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:17:25.402713+00:00', 'runtime': '865320.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.988', 'adc_vl_f': '14.404', 'adc_ic_f': '1.669', 'adc_il_f': '0.362', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.067', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865324418 ms)
2025-02-17 20:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:16:25.400061+00:00', 'runtime': '865260.001', 'adc_vb_f': '14.423', 'adc_va_f': '22.095', 'adc_vl_f': '14.377', 'adc_ic_f': '1.657', 'adc_il_f': '0.358', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.856', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865324418 ms)
2025-02-17 20:17:35.761 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865333580 ms)
2025-02-17 20:17:35.762 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865333581 ms)
2025-02-17 20:17:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:17:25.400083+00:00'} (T+865333582 ms)
2025-02-17 20:18:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865384416 ms)
2025-02-17 20:18:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865384416 ms)
2025-02-17 20:18:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:18:25.402656+00:00', 'runtime': '865380.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.988', 'adc_vl_f': '14.404', 'adc_ic_f': '1.679', 'adc_il_f': '0.355', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.4', 'ahc_t': '2656.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.203', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.4', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865384420 ms)
2025-02-17 20:18:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:17:25.402713+00:00', 'runtime': '865320.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.988', 'adc_vl_f': '14.404', 'adc_ic_f': '1.669', 'adc_il_f': '0.362', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.3', 'ahc_t': '2656.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.067', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.3', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865384421 ms)
2025-02-17 20:18:35.683 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865393502 ms)
2025-02-17 20:18:35.683 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865393503 ms)
2025-02-17 20:18:35.684 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:18:25.400036+00:00'} (T+865393503 ms)
2025-02-17 20:19:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865444415 ms)
2025-02-17 20:19:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865444416 ms)
2025-02-17 20:19:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:19:25.403072+00:00', 'runtime': '865440.004', 'adc_vb_f': '14.413', 'adc_va_f': '21.954', 'adc_vl_f': '14.38', 'adc_ic_f': '1.517', 'adc_il_f': '0.365', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.4', 'ahc_t': '2656.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.916', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.4', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865444420 ms)
2025-02-17 20:19:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:18:25.402656+00:00', 'runtime': '865380.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.988', 'adc_vl_f': '14.404', 'adc_ic_f': '1.679', 'adc_il_f': '0.355', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.4', 'ahc_t': '2656.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.203', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.4', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865444421 ms)
2025-02-17 20:19:35.702 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865453521 ms)
2025-02-17 20:19:35.703 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865453522 ms)
2025-02-17 20:19:35.704 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:19:25.400425+00:00'} (T+865453523 ms)
2025-02-17 20:20:05.313 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+865483133 ms)
2025-02-17 20:20:05.314 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+865483133 ms)
2025-02-17 20:20:05.314 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59528)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+865483133 ms)
2025-02-17 20:20:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865504420 ms)
2025-02-17 20:20:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865504420 ms)
2025-02-17 20:20:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:20:25.402725+00:00', 'runtime': '865500.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.881', 'adc_vl_f': '14.395', 'adc_ic_f': '1.662', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.4', 'ahc_t': '2656.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.901', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.4', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865504423 ms)
2025-02-17 20:20:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:19:25.403072+00:00', 'runtime': '865440.004', 'adc_vb_f': '14.413', 'adc_va_f': '21.954', 'adc_vl_f': '14.38', 'adc_ic_f': '1.517', 'adc_il_f': '0.365', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.4', 'ahc_t': '2656.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.916', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.4', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865504424 ms)
2025-02-17 20:20:35.750 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865513569 ms)
2025-02-17 20:20:35.751 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865513570 ms)
2025-02-17 20:20:35.752 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:20:25.400106+00:00'} (T+865513571 ms)
2025-02-17 20:21:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865564410 ms)
2025-02-17 20:21:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865564411 ms)
2025-02-17 20:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:21:25.400052+00:00', 'runtime': '865560.001', 'adc_vb_f': '14.423', 'adc_va_f': '21.878', 'adc_vl_f': '14.389', 'adc_ic_f': '1.66', 'adc_il_f': '0.321', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.901', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865564415 ms)
2025-02-17 20:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:20:25.402725+00:00', 'runtime': '865500.003', 'adc_vb_f': '14.423', 'adc_va_f': '21.881', 'adc_vl_f': '14.395', 'adc_ic_f': '1.662', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.4', 'ahc_t': '2656.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.901', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.4', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865564416 ms)
2025-02-17 20:21:29.727 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+865567546 ms)
2025-02-17 20:21:29.727 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+865567546 ms)
2025-02-17 20:21:29.728 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59618)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+865567547 ms)
2025-02-17 20:22:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865624410 ms)
2025-02-17 20:22:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865624411 ms)
2025-02-17 20:22:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:22:25.400001+00:00', 'runtime': '865620.001', 'adc_vb_f': '14.423', 'adc_va_f': '21.878', 'adc_vl_f': '14.377', 'adc_ic_f': '1.655', 'adc_il_f': '0.312', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.931', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865624415 ms)
2025-02-17 20:22:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:21:25.400052+00:00', 'runtime': '865560.001', 'adc_vb_f': '14.423', 'adc_va_f': '21.878', 'adc_vl_f': '14.389', 'adc_ic_f': '1.66', 'adc_il_f': '0.321', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.423', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.901', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865624415 ms)
2025-02-17 20:23:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865684414 ms)
2025-02-17 20:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865684414 ms)
2025-02-17 20:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:23:25.400242+00:00', 'runtime': '865680.001', 'adc_vb_f': '14.423', 'adc_va_f': '21.878', 'adc_vl_f': '14.377', 'adc_ic_f': '1.648', 'adc_il_f': '0.35', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.856', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865684418 ms)
2025-02-17 20:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:22:25.400001+00:00', 'runtime': '865620.001', 'adc_vb_f': '14.423', 'adc_va_f': '21.878', 'adc_vl_f': '14.377', 'adc_ic_f': '1.655', 'adc_il_f': '0.312', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.931', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865684419 ms)
2025-02-17 20:23:35.646 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865693466 ms)
2025-02-17 20:23:35.647 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865693466 ms)
2025-02-17 20:23:35.648 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:23:25.413055+00:00'} (T+865693467 ms)
2025-02-17 20:24:25.647 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865743467 ms)
2025-02-17 20:24:25.648 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865743467 ms)
2025-02-17 20:24:25.652 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:24:25.401120+00:00', 'runtime': '865740.002', 'adc_vb_f': '14.423', 'adc_va_f': '21.881', 'adc_vl_f': '14.377', 'adc_ic_f': '1.602', 'adc_il_f': '0.333', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.539', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865743471 ms)
2025-02-17 20:24:25.652 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:23:25.400242+00:00', 'runtime': '865680.001', 'adc_vb_f': '14.423', 'adc_va_f': '21.878', 'adc_vl_f': '14.377', 'adc_ic_f': '1.648', 'adc_il_f': '0.35', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.856', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865743472 ms)
2025-02-17 20:25:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865804407 ms)
2025-02-17 20:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865804408 ms)
2025-02-17 20:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:25:25.400743+00:00', 'runtime': '865800.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.936', 'adc_vl_f': '14.377', 'adc_ic_f': '1.539', 'adc_il_f': '0.36', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.392', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.059', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865804412 ms)
2025-02-17 20:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:24:25.401120+00:00', 'runtime': '865740.002', 'adc_vb_f': '14.423', 'adc_va_f': '21.881', 'adc_vl_f': '14.377', 'adc_ic_f': '1.602', 'adc_il_f': '0.333', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.42', 'vb_ref': '13.942', 'ahc_r': '2656.5', 'ahc_t': '2656.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.539', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.5', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865804412 ms)
2025-02-17 20:25:33.111 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+865810930 ms)
2025-02-17 20:25:33.111 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+865810930 ms)
2025-02-17 20:25:33.112 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59876)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+865810931 ms)
2025-02-17 20:25:41.205 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+865819024 ms)
2025-02-17 20:25:41.205 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+865819025 ms)
2025-02-17 20:25:41.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59886)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+865819025 ms)
2025-02-17 20:26:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865864411 ms)
2025-02-17 20:26:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865864412 ms)
2025-02-17 20:26:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:26:25.400185+00:00', 'runtime': '865860.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.988', 'adc_vl_f': '14.377', 'adc_ic_f': '1.573', 'adc_il_f': '0.37', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.587', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865864413 ms)
2025-02-17 20:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:25:25.400743+00:00', 'runtime': '865800.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.936', 'adc_vl_f': '14.377', 'adc_ic_f': '1.539', 'adc_il_f': '0.36', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.392', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.059', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865864413 ms)
2025-02-17 20:27:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865924414 ms)
2025-02-17 20:27:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865924415 ms)
2025-02-17 20:27:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:27:25.400190+00:00', 'runtime': '865920.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.988', 'adc_vl_f': '14.377', 'adc_ic_f': '1.51', 'adc_il_f': '0.345', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.606', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865924419 ms)
2025-02-17 20:27:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:26:25.400185+00:00', 'runtime': '865860.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.988', 'adc_vl_f': '14.377', 'adc_ic_f': '1.573', 'adc_il_f': '0.37', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.587', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865924419 ms)
2025-02-17 20:27:30.433 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+865928252 ms)
2025-02-17 20:27:30.433 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+865928252 ms)
2025-02-17 20:27:30.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60010)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+865928253 ms)
2025-02-17 20:27:35.892 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865933712 ms)
2025-02-17 20:27:35.893 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865933712 ms)
2025-02-17 20:27:35.894 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:27:25.413088+00:00'} (T+865933713 ms)
2025-02-17 20:28:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+865984419 ms)
2025-02-17 20:28:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+865984419 ms)
2025-02-17 20:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:28:25.402839+00:00', 'runtime': '865980.003', 'adc_vb_f': '14.389', 'adc_va_f': '21.985', 'adc_vl_f': '14.377', 'adc_ic_f': '1.599', 'adc_il_f': '0.329', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.346', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865984423 ms)
2025-02-17 20:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:27:25.400190+00:00', 'runtime': '865920.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.988', 'adc_vl_f': '14.377', 'adc_ic_f': '1.51', 'adc_il_f': '0.345', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.606', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+865984424 ms)
2025-02-17 20:28:35.765 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865993585 ms)
2025-02-17 20:28:35.766 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+865993585 ms)
2025-02-17 20:28:35.767 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:28:25.400228+00:00'} (T+865993587 ms)
2025-02-17 20:29:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866044408 ms)
2025-02-17 20:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866044408 ms)
2025-02-17 20:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:29:25.402584+00:00', 'runtime': '866040.003', 'adc_vb_f': '14.386', 'adc_va_f': '22.018', 'adc_vl_f': '14.377', 'adc_ic_f': '1.522', 'adc_il_f': '0.319', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.576', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866044412 ms)
2025-02-17 20:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:28:25.402839+00:00', 'runtime': '865980.003', 'adc_vb_f': '14.389', 'adc_va_f': '21.985', 'adc_vl_f': '14.377', 'adc_ic_f': '1.599', 'adc_il_f': '0.329', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.6', 'ahc_t': '2656.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.346', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.6', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866044413 ms)
2025-02-17 20:29:36.406 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866054225 ms)
2025-02-17 20:29:36.406 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866054226 ms)
2025-02-17 20:29:36.408 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:29:25.400027+00:00'} (T+866054227 ms)
2025-02-17 20:29:58.696 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866076515 ms)
2025-02-17 20:29:58.696 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866076515 ms)
2025-02-17 20:29:58.697 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60172)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866076516 ms)
2025-02-17 20:30:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866104408 ms)
2025-02-17 20:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866104408 ms)
2025-02-17 20:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:30:25.402898+00:00', 'runtime': '866100.003', 'adc_vb_f': '14.386', 'adc_va_f': '22.049', 'adc_vl_f': '14.377', 'adc_ic_f': '1.59', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.377', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866104411 ms)
2025-02-17 20:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:29:25.402584+00:00', 'runtime': '866040.003', 'adc_vb_f': '14.386', 'adc_va_f': '22.018', 'adc_vl_f': '14.377', 'adc_ic_f': '1.522', 'adc_il_f': '0.319', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.576', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866104412 ms)
2025-02-17 20:30:35.954 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866113774 ms)
2025-02-17 20:30:35.955 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866113774 ms)
2025-02-17 20:30:35.956 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:30:25.400272+00:00'} (T+866113775 ms)
2025-02-17 20:31:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866164410 ms)
2025-02-17 20:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866164410 ms)
2025-02-17 20:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:31:25.400510+00:00', 'runtime': '866160.001', 'adc_vb_f': '14.386', 'adc_va_f': '22.083', 'adc_vl_f': '14.377', 'adc_ic_f': '1.582', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.693', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866164412 ms)
2025-02-17 20:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:30:25.402898+00:00', 'runtime': '866100.003', 'adc_vb_f': '14.386', 'adc_va_f': '22.049', 'adc_vl_f': '14.377', 'adc_ic_f': '1.59', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.377', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866164413 ms)
2025-02-17 20:31:35.892 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866173712 ms)
2025-02-17 20:31:35.893 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866173712 ms)
2025-02-17 20:31:35.894 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:31:25.413146+00:00'} (T+866173713 ms)
2025-02-17 20:32:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866224412 ms)
2025-02-17 20:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866224413 ms)
2025-02-17 20:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:32:25.400170+00:00', 'runtime': '866220.001', 'adc_vb_f': '14.386', 'adc_va_f': '22.018', 'adc_vl_f': '14.377', 'adc_ic_f': '1.532', 'adc_il_f': '0.338', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.059', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866224415 ms)
2025-02-17 20:32:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:31:25.400510+00:00', 'runtime': '866160.001', 'adc_vb_f': '14.386', 'adc_va_f': '22.083', 'adc_vl_f': '14.377', 'adc_ic_f': '1.582', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.693', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866224415 ms)
2025-02-17 20:33:06.007 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866263826 ms)
2025-02-17 20:33:06.007 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866263826 ms)
2025-02-17 20:33:06.007 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60372)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866263827 ms)
2025-02-17 20:33:24.076 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866281896 ms)
2025-02-17 20:33:24.077 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866281896 ms)
2025-02-17 20:33:24.077 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60392)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866281897 ms)
2025-02-17 20:33:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866284415 ms)
2025-02-17 20:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866284415 ms)
2025-02-17 20:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:33:25.400172+00:00', 'runtime': '866280.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.985', 'adc_vl_f': '14.374', 'adc_ic_f': '1.621', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.8', 'ahc_t': '2656.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.313', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.8', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866284416 ms)
2025-02-17 20:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:32:25.400170+00:00', 'runtime': '866220.001', 'adc_vb_f': '14.386', 'adc_va_f': '22.018', 'adc_vl_f': '14.377', 'adc_ic_f': '1.532', 'adc_il_f': '0.338', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.7', 'ahc_t': '2656.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.059', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.7', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866284417 ms)
2025-02-17 20:34:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866344402 ms)
2025-02-17 20:34:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866344402 ms)
2025-02-17 20:34:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:34:25.400614+00:00', 'runtime': '866340.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.988', 'adc_vl_f': '14.377', 'adc_ic_f': '1.51', 'adc_il_f': '0.355', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.8', 'ahc_t': '2656.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.682', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.8', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866344404 ms)
2025-02-17 20:34:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:33:25.400172+00:00', 'runtime': '866280.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.985', 'adc_vl_f': '14.374', 'adc_ic_f': '1.621', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.8', 'ahc_t': '2656.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.313', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.8', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866344404 ms)
2025-02-17 20:35:03.297 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866381117 ms)
2025-02-17 20:35:03.298 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866381117 ms)
2025-02-17 20:35:03.298 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60496)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866381117 ms)
2025-02-17 20:35:25.395 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866403214 ms)
2025-02-17 20:35:25.395 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866403214 ms)
2025-02-17 20:35:25.396 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60520)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866403215 ms)
2025-02-17 20:35:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866404420 ms)
2025-02-17 20:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866404421 ms)
2025-02-17 20:35:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:35:25.402838+00:00', 'runtime': '866400.003', 'adc_vb_f': '14.386', 'adc_va_f': '21.896', 'adc_vl_f': '14.377', 'adc_ic_f': '1.522', 'adc_il_f': '0.35', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.8', 'ahc_t': '2656.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.817', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.8', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866404422 ms)
2025-02-17 20:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:34:25.400614+00:00', 'runtime': '866340.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.988', 'adc_vl_f': '14.377', 'adc_ic_f': '1.51', 'adc_il_f': '0.355', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.8', 'ahc_t': '2656.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.682', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.8', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866404422 ms)
2025-02-17 20:35:35.886 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866413706 ms)
2025-02-17 20:35:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866413706 ms)
2025-02-17 20:35:35.888 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:35:25.400166+00:00'} (T+866413707 ms)
2025-02-17 20:35:40.506 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866418326 ms)
2025-02-17 20:35:40.507 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866418326 ms)
2025-02-17 20:35:40.508 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60538)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866418327 ms)
2025-02-17 20:36:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866464409 ms)
2025-02-17 20:36:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866464410 ms)
2025-02-17 20:36:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:36:25.402821+00:00', 'runtime': '866460.003', 'adc_vb_f': '14.386', 'adc_va_f': '21.991', 'adc_vl_f': '14.377', 'adc_ic_f': '1.565', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.436', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866464412 ms)
2025-02-17 20:36:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:35:25.402838+00:00', 'runtime': '866400.003', 'adc_vb_f': '14.386', 'adc_va_f': '21.896', 'adc_vl_f': '14.377', 'adc_ic_f': '1.522', 'adc_il_f': '0.35', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2656.8', 'ahc_t': '2656.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.0', 'ahl_t': '1592.0', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.817', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.8', 'ahl_daily': '7.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866464412 ms)
2025-02-17 20:36:36.067 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866473886 ms)
2025-02-17 20:36:36.068 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866473887 ms)
2025-02-17 20:36:36.069 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:36:25.400207+00:00'} (T+866473888 ms)
2025-02-17 20:37:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866524408 ms)
2025-02-17 20:37:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866524409 ms)
2025-02-17 20:37:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:37:25.402871+00:00', 'runtime': '866520.003', 'adc_vb_f': '14.346', 'adc_va_f': '21.994', 'adc_vl_f': '14.34', 'adc_ic_f': '1.551', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.352', 'vb_ref': '13.883', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.21', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866524411 ms)
2025-02-17 20:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:36:25.402821+00:00', 'runtime': '866460.003', 'adc_vb_f': '14.386', 'adc_va_f': '21.991', 'adc_vl_f': '14.377', 'adc_ic_f': '1.565', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.436', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866524411 ms)
2025-02-17 20:38:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866584410 ms)
2025-02-17 20:38:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866584411 ms)
2025-02-17 20:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:38:25.400175+00:00', 'runtime': '866580.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.994', 'adc_vl_f': '14.34', 'adc_ic_f': '1.551', 'adc_il_f': '0.312', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.24', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866584413 ms)
2025-02-17 20:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:37:25.402871+00:00', 'runtime': '866520.003', 'adc_vb_f': '14.346', 'adc_va_f': '21.994', 'adc_vl_f': '14.34', 'adc_ic_f': '1.551', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.352', 'vb_ref': '13.883', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.21', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866584413 ms)
2025-02-17 20:38:57.858 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866615678 ms)
2025-02-17 20:38:57.859 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866615678 ms)
2025-02-17 20:38:57.859 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60758)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866615678 ms)
2025-02-17 20:39:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866644414 ms)
2025-02-17 20:39:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866644415 ms)
2025-02-17 20:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:39:25.400153+00:00', 'runtime': '866640.001', 'adc_vb_f': '14.355', 'adc_va_f': '22.095', 'adc_vl_f': '14.34', 'adc_ic_f': '1.573', 'adc_il_f': '0.329', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.316', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866644417 ms)
2025-02-17 20:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:38:25.400175+00:00', 'runtime': '866580.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.994', 'adc_vl_f': '14.34', 'adc_ic_f': '1.551', 'adc_il_f': '0.312', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.24', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866644417 ms)
2025-02-17 20:39:30.968 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866648787 ms)
2025-02-17 20:39:30.968 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866648788 ms)
2025-02-17 20:39:30.969 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60792)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866648788 ms)
2025-02-17 20:39:35.789 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866653608 ms)
2025-02-17 20:39:35.790 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866653609 ms)
2025-02-17 20:39:35.791 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:39:25.409046+00:00'} (T+866653610 ms)
2025-02-17 20:40:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866704418 ms)
2025-02-17 20:40:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866704418 ms)
2025-02-17 20:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:40:25.402580+00:00', 'runtime': '866700.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.116', 'adc_vl_f': '14.34', 'adc_ic_f': '1.536', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.999', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866704420 ms)
2025-02-17 20:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:39:25.400153+00:00', 'runtime': '866640.001', 'adc_vb_f': '14.355', 'adc_va_f': '22.095', 'adc_vl_f': '14.34', 'adc_ic_f': '1.573', 'adc_il_f': '0.329', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2656.9', 'ahc_t': '2656.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.316', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '26.9', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866704421 ms)
2025-02-17 20:40:35.918 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866713738 ms)
2025-02-17 20:40:35.919 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866713738 ms)
2025-02-17 20:40:35.920 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:40:25.400004+00:00'} (T+866713739 ms)
2025-02-17 20:41:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866764420 ms)
2025-02-17 20:41:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866764421 ms)
2025-02-17 20:41:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:41:25.402786+00:00', 'runtime': '866760.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.162', 'adc_vl_f': '14.34', 'adc_ic_f': '1.524', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.893', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866764423 ms)
2025-02-17 20:41:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:40:25.402580+00:00', 'runtime': '866700.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.116', 'adc_vl_f': '14.34', 'adc_ic_f': '1.536', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.999', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866764423 ms)
2025-02-17 20:41:35.730 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866773550 ms)
2025-02-17 20:41:35.731 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+866773550 ms)
2025-02-17 20:41:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:41:25.400203+00:00'} (T+866773551 ms)
2025-02-17 20:42:04.250 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+866802069 ms)
2025-02-17 20:42:04.250 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+866802069 ms)
2025-02-17 20:42:04.250 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60958)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+866802069 ms)
2025-02-17 20:42:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866824423 ms)
2025-02-17 20:42:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866824424 ms)
2025-02-17 20:42:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:42:25.400368+00:00', 'runtime': '866820.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.539', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.104', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866824425 ms)
2025-02-17 20:42:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:41:25.402786+00:00', 'runtime': '866760.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.162', 'adc_vl_f': '14.34', 'adc_ic_f': '1.524', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.893', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866824425 ms)
2025-02-17 20:43:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866884407 ms)
2025-02-17 20:43:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866884408 ms)
2025-02-17 20:43:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:43:25.400190+00:00', 'runtime': '866880.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.539', 'adc_il_f': '0.336', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.104', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866884409 ms)
2025-02-17 20:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:42:25.400368+00:00', 'runtime': '866820.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.539', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.104', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866884409 ms)
2025-02-17 20:44:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+866944409 ms)
2025-02-17 20:44:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+866944410 ms)
2025-02-17 20:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:44:25.400264+00:00', 'runtime': '866940.001', 'adc_vb_f': '14.355', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.505', 'adc_il_f': '0.316', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.515', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866944411 ms)
2025-02-17 20:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:43:25.400190+00:00', 'runtime': '866880.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.539', 'adc_il_f': '0.336', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.0', 'ahc_t': '2657.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.104', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.0', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+866944411 ms)
2025-02-17 20:45:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867004415 ms)
2025-02-17 20:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867004416 ms)
2025-02-17 20:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:45:25.400260+00:00', 'runtime': '867000.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.116', 'adc_vl_f': '14.34', 'adc_ic_f': '1.532', 'adc_il_f': '0.319', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.968', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867004417 ms)
2025-02-17 20:45:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:44:25.400264+00:00', 'runtime': '866940.001', 'adc_vb_f': '14.355', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.505', 'adc_il_f': '0.316', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.515', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867004417 ms)
2025-02-17 20:46:26.581 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867064401 ms)
2025-02-17 20:46:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867064401 ms)
2025-02-17 20:46:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:46:25.400163+00:00', 'runtime': '867060.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.095', 'adc_vl_f': '14.34', 'adc_ic_f': '1.493', 'adc_il_f': '0.316', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.44', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867064402 ms)
2025-02-17 20:46:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:45:25.400260+00:00', 'runtime': '867000.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.116', 'adc_vl_f': '14.34', 'adc_ic_f': '1.532', 'adc_il_f': '0.319', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.968', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867064403 ms)
2025-02-17 20:47:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867124404 ms)
2025-02-17 20:47:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867124405 ms)
2025-02-17 20:47:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:47:25.400188+00:00', 'runtime': '867120.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.198', 'adc_vl_f': '14.34', 'adc_ic_f': '1.527', 'adc_il_f': '0.312', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.893', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867124406 ms)
2025-02-17 20:47:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:46:25.400163+00:00', 'runtime': '867060.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.095', 'adc_vl_f': '14.34', 'adc_ic_f': '1.493', 'adc_il_f': '0.316', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.44', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867124406 ms)
2025-02-17 20:48:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867184424 ms)
2025-02-17 20:48:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867184425 ms)
2025-02-17 20:48:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:48:25.400209+00:00', 'runtime': '867180.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.522', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.833', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867184429 ms)
2025-02-17 20:48:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:47:25.400188+00:00', 'runtime': '867120.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.198', 'adc_vl_f': '14.34', 'adc_ic_f': '1.527', 'adc_il_f': '0.312', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.1', 'ahc_t': '2657.1', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.893', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.1', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867184429 ms)
2025-02-17 20:48:35.670 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867193490 ms)
2025-02-17 20:48:35.671 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867193490 ms)
2025-02-17 20:48:35.672 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:48:25.413048+00:00'} (T+867193492 ms)
2025-02-17 20:49:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867244410 ms)
2025-02-17 20:49:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867244411 ms)
2025-02-17 20:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:49:25.401117+00:00', 'runtime': '867240.002', 'adc_vb_f': '14.346', 'adc_va_f': '22.284', 'adc_vl_f': '14.34', 'adc_ic_f': '1.522', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.863', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867244415 ms)
2025-02-17 20:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:48:25.400209+00:00', 'runtime': '867180.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.522', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.833', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867244415 ms)
2025-02-17 20:49:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867253420 ms)
2025-02-17 20:49:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867253421 ms)
2025-02-17 20:49:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:49:25.400099+00:00'} (T+867253423 ms)
2025-02-17 20:50:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867304410 ms)
2025-02-17 20:50:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867304411 ms)
2025-02-17 20:50:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:50:25.400165+00:00', 'runtime': '867300.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.505', 'adc_il_f': '0.345', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.591', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867304415 ms)
2025-02-17 20:50:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:49:25.401117+00:00', 'runtime': '867240.002', 'adc_vb_f': '14.346', 'adc_va_f': '22.284', 'adc_vl_f': '14.34', 'adc_ic_f': '1.522', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.863', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867304415 ms)
2025-02-17 20:50:35.770 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867313589 ms)
2025-02-17 20:50:35.770 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867313590 ms)
2025-02-17 20:50:35.771 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:50:25.405137+00:00'} (T+867313591 ms)
2025-02-17 20:51:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867364412 ms)
2025-02-17 20:51:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867364413 ms)
2025-02-17 20:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:51:25.402652+00:00', 'runtime': '867360.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.476', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.198', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867364417 ms)
2025-02-17 20:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:50:25.400165+00:00', 'runtime': '867300.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.505', 'adc_il_f': '0.345', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.591', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867364417 ms)
2025-02-17 20:52:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867424412 ms)
2025-02-17 20:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867424412 ms)
2025-02-17 20:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:52:25.400008+00:00', 'runtime': '867420.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.507', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.591', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867424416 ms)
2025-02-17 20:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:51:25.402652+00:00', 'runtime': '867360.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.476', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.2', 'ahc_t': '2657.2', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.198', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.2', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867424417 ms)
2025-02-17 20:53:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867484415 ms)
2025-02-17 20:53:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867484416 ms)
2025-02-17 20:53:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:53:25.400170+00:00', 'runtime': '867480.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.51', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.651', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867484418 ms)
2025-02-17 20:53:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:52:25.400008+00:00', 'runtime': '867420.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.507', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.591', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867484419 ms)
2025-02-17 20:53:35.631 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867493450 ms)
2025-02-17 20:53:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867493450 ms)
2025-02-17 20:53:35.632 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:53:25.400330+00:00'} (T+867493452 ms)
2025-02-17 20:54:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867544417 ms)
2025-02-17 20:54:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867544418 ms)
2025-02-17 20:54:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:54:25.400179+00:00', 'runtime': '867540.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.488', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.75', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867544421 ms)
2025-02-17 20:54:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:53:25.400170+00:00', 'runtime': '867480.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.51', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.1', 'ahl_t': '1592.1', 'hourmeter': '20127', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.651', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '7.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867544422 ms)
2025-02-17 20:54:35.754 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867553573 ms)
2025-02-17 20:54:35.755 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867553574 ms)
2025-02-17 20:54:35.756 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:54:25.400179+00:00'} (T+867553575 ms)
2025-02-17 20:55:25.313 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+867603132 ms)
2025-02-17 20:55:25.313 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+867603132 ms)
2025-02-17 20:55:25.313 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33580)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+867603133 ms)
2025-02-17 20:55:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867604405 ms)
2025-02-17 20:55:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867604406 ms)
2025-02-17 20:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:55:25.400186+00:00', 'runtime': '867600.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.299', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867604407 ms)
2025-02-17 20:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:54:25.400179+00:00', 'runtime': '867540.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.488', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.75', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867604409 ms)
2025-02-17 20:55:35.406 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867613225 ms)
2025-02-17 20:55:35.407 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867613226 ms)
2025-02-17 20:55:35.408 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:55:25.400122+00:00'} (T+867613227 ms)
2025-02-17 20:56:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867664409 ms)
2025-02-17 20:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867664409 ms)
2025-02-17 20:56:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:56:25.400176+00:00', 'runtime': '867660.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.459', 'adc_il_f': '0.35', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.851', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867664410 ms)
2025-02-17 20:56:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:55:25.400186+00:00', 'runtime': '867600.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.299', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.3', 'ahc_t': '2657.3', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.3', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867664412 ms)
2025-02-17 20:57:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867724410 ms)
2025-02-17 20:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867724411 ms)
2025-02-17 20:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:57:25.400195+00:00', 'runtime': '867720.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.205', 'adc_vl_f': '14.34', 'adc_ic_f': '1.5', 'adc_il_f': '0.307', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.485', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867724412 ms)
2025-02-17 20:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:56:25.400176+00:00', 'runtime': '867660.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.459', 'adc_il_f': '0.35', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.851', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867724412 ms)
2025-02-17 20:58:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867784414 ms)
2025-02-17 20:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867784415 ms)
2025-02-17 20:58:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:58:25.400259+00:00', 'runtime': '867780.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.205', 'adc_vl_f': '14.34', 'adc_ic_f': '1.503', 'adc_il_f': '0.319', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.561', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867784416 ms)
2025-02-17 20:58:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:57:25.400195+00:00', 'runtime': '867720.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.205', 'adc_vl_f': '14.34', 'adc_ic_f': '1.5', 'adc_il_f': '0.307', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.485', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867784416 ms)
2025-02-17 20:59:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867844416 ms)
2025-02-17 20:59:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867844416 ms)
2025-02-17 20:59:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 20:59:25.400171+00:00', 'runtime': '867840.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.491', 'adc_il_f': '0.348', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.38', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867844418 ms)
2025-02-17 20:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:58:25.400259+00:00', 'runtime': '867780.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.205', 'adc_vl_f': '14.34', 'adc_ic_f': '1.503', 'adc_il_f': '0.319', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.561', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867844418 ms)
2025-02-17 20:59:35.917 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867853736 ms)
2025-02-17 20:59:35.918 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+867853737 ms)
2025-02-17 20:59:35.919 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 20:59:25.400340+00:00'} (T+867853738 ms)
2025-02-17 20:59:43.682 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+867861501 ms)
2025-02-17 20:59:43.682 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+867861502 ms)
2025-02-17 20:59:43.683 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33864)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+867861502 ms)
2025-02-17 21:00:05.781 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+867883600 ms)
2025-02-17 21:00:05.781 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+867883600 ms)
2025-02-17 21:00:05.781 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33890)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+867883600 ms)
2025-02-17 21:00:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867904419 ms)
2025-02-17 21:00:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867904419 ms)
2025-02-17 21:00:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:00:25.400167+00:00', 'runtime': '867900.001', 'adc_vb_f': '14.359', 'adc_va_f': '22.272', 'adc_vl_f': '14.34', 'adc_ic_f': '1.575', 'adc_il_f': '0.338', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.727', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867904423 ms)
2025-02-17 21:00:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 20:59:25.400171+00:00', 'runtime': '867840.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.491', 'adc_il_f': '0.348', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.4', 'ahc_t': '2657.4', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.38', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.4', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867904423 ms)
2025-02-17 21:01:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+867964405 ms)
2025-02-17 21:01:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+867964405 ms)
2025-02-17 21:01:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:01:25.400777+00:00', 'runtime': '867960.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.302', 'adc_vl_f': '14.34', 'adc_ic_f': '1.488', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.304', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867964406 ms)
2025-02-17 21:01:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:00:25.400167+00:00', 'runtime': '867900.001', 'adc_vb_f': '14.359', 'adc_va_f': '22.272', 'adc_vl_f': '14.34', 'adc_ic_f': '1.575', 'adc_il_f': '0.338', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.727', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+867964407 ms)
2025-02-17 21:01:31.952 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+867969771 ms)
2025-02-17 21:01:31.952 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+867969771 ms)
2025-02-17 21:01:31.952 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33982)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+867969772 ms)
2025-02-17 21:02:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868024406 ms)
2025-02-17 21:02:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868024407 ms)
2025-02-17 21:02:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:02:25.400186+00:00', 'runtime': '868020.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.481', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.198', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868024408 ms)
2025-02-17 21:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:01:25.400777+00:00', 'runtime': '867960.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.302', 'adc_vl_f': '14.34', 'adc_ic_f': '1.488', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.304', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868024408 ms)
2025-02-17 21:03:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868084409 ms)
2025-02-17 21:03:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868084409 ms)
2025-02-17 21:03:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:03:25.400192+00:00', 'runtime': '868080.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.284', 'adc_vl_f': '14.34', 'adc_ic_f': '1.478', 'adc_il_f': '0.358', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.274', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868084410 ms)
2025-02-17 21:03:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:02:25.400186+00:00', 'runtime': '868020.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.481', 'adc_il_f': '0.365', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.198', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868084410 ms)
2025-02-17 21:03:44.182 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+868102001 ms)
2025-02-17 21:03:44.182 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+868102002 ms)
2025-02-17 21:03:44.183 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34122)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868102002 ms)
2025-02-17 21:04:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868144412 ms)
2025-02-17 21:04:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868144412 ms)
2025-02-17 21:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:04:25.400598+00:00', 'runtime': '868140.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.338', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.093', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868144416 ms)
2025-02-17 21:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:03:25.400192+00:00', 'runtime': '868080.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.284', 'adc_vl_f': '14.34', 'adc_ic_f': '1.478', 'adc_il_f': '0.358', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.5', 'ahc_t': '2657.5', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.274', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.5', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868144417 ms)
2025-02-17 21:04:52.310 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+868170130 ms)
2025-02-17 21:04:52.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+868170130 ms)
2025-02-17 21:04:52.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34196)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868170130 ms)
2025-02-17 21:05:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868204415 ms)
2025-02-17 21:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868204416 ms)
2025-02-17 21:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:05:25.400683+00:00', 'runtime': '868200.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.488', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.334', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868204420 ms)
2025-02-17 21:05:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:04:25.400598+00:00', 'runtime': '868140.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.338', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.093', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868204420 ms)
2025-02-17 21:06:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868264417 ms)
2025-02-17 21:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868264417 ms)
2025-02-17 21:06:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:06:25.402767+00:00', 'runtime': '868260.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868264420 ms)
2025-02-17 21:06:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:05:25.400683+00:00', 'runtime': '868200.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.488', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.334', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868264421 ms)
2025-02-17 21:07:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868324403 ms)
2025-02-17 21:07:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868324403 ms)
2025-02-17 21:07:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:07:25.400642+00:00', 'runtime': '868320.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.269', 'adc_vl_f': '14.34', 'adc_ic_f': '1.464', 'adc_il_f': '0.304', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.957', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868324407 ms)
2025-02-17 21:07:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:06:25.402767+00:00', 'runtime': '868260.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.308', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868324408 ms)
2025-02-17 21:08:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868384407 ms)
2025-02-17 21:08:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868384407 ms)
2025-02-17 21:08:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:08:25.400189+00:00', 'runtime': '868380.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.454', 'adc_il_f': '0.336', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868384411 ms)
2025-02-17 21:08:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:07:25.400642+00:00', 'runtime': '868320.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.269', 'adc_vl_f': '14.34', 'adc_ic_f': '1.464', 'adc_il_f': '0.304', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.6', 'ahc_t': '2657.6', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.957', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.6', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868384411 ms)
2025-02-17 21:09:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868444408 ms)
2025-02-17 21:09:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868444409 ms)
2025-02-17 21:09:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:09:25.400200+00:00', 'runtime': '868440.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.312', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.108', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868444413 ms)
2025-02-17 21:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:08:25.400189+00:00', 'runtime': '868380.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.454', 'adc_il_f': '0.336', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868444413 ms)
2025-02-17 21:10:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868504414 ms)
2025-02-17 21:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868504415 ms)
2025-02-17 21:10:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:10:25.400171+00:00', 'runtime': '868500.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.319', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868504419 ms)
2025-02-17 21:10:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:09:25.400200+00:00', 'runtime': '868440.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.312', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.108', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868504419 ms)
2025-02-17 21:11:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868564418 ms)
2025-02-17 21:11:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868564419 ms)
2025-02-17 21:11:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:11:25.402892+00:00', 'runtime': '868560.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.459', 'adc_il_f': '0.35', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.927', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868564423 ms)
2025-02-17 21:11:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:10:25.400171+00:00', 'runtime': '868500.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.474', 'adc_il_f': '0.319', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868564423 ms)
2025-02-17 21:11:35.953 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868573772 ms)
2025-02-17 21:11:35.954 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868573773 ms)
2025-02-17 21:11:35.955 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:11:25.400199+00:00'} (T+868573774 ms)
2025-02-17 21:12:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868624419 ms)
2025-02-17 21:12:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868624420 ms)
2025-02-17 21:12:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:12:25.400158+00:00', 'runtime': '868620.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.113', 'adc_vl_f': '14.34', 'adc_ic_f': '1.466', 'adc_il_f': '0.341', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.032', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868624424 ms)
2025-02-17 21:12:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:11:25.402892+00:00', 'runtime': '868560.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.459', 'adc_il_f': '0.35', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.7', 'ahc_t': '2657.7', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.927', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.7', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868624424 ms)
2025-02-17 21:12:35.864 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868633684 ms)
2025-02-17 21:12:35.865 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868633684 ms)
2025-02-17 21:12:35.866 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:12:25.413122+00:00'} (T+868633685 ms)
2025-02-17 21:13:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868684406 ms)
2025-02-17 21:13:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868684406 ms)
2025-02-17 21:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:13:25.402837+00:00', 'runtime': '868680.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.131', 'adc_vl_f': '14.34', 'adc_ic_f': '1.437', 'adc_il_f': '0.338', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.817', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868684410 ms)
2025-02-17 21:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:12:25.400158+00:00', 'runtime': '868620.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.113', 'adc_vl_f': '14.34', 'adc_ic_f': '1.466', 'adc_il_f': '0.341', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.2', 'ahl_t': '1592.2', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.032', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868684410 ms)
2025-02-17 21:13:32.973 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868690793 ms)
2025-02-17 21:13:32.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868690793 ms)
2025-02-17 21:13:32.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868690793 ms)
2025-02-17 21:13:32.975 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868690794 ms)
2025-02-17 21:13:32.975 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868690794 ms)
2025-02-17 21:13:32.975 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868690795 ms)
2025-02-17 21:13:32.976 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868690795 ms)
2025-02-17 21:13:32.976 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868690795 ms)
2025-02-17 21:13:32.976 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868690796 ms)
2025-02-17 21:13:32.977 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868690796 ms)
2025-02-17 21:13:32.977 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868690796 ms)
2025-02-17 21:13:32.978 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868690797 ms)
2025-02-17 21:13:32.978 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868690797 ms)
2025-02-17 21:13:32.978 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868690797 ms)
2025-02-17 21:13:32.978 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868690798 ms)
2025-02-17 21:13:32.979 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868690798 ms)
2025-02-17 21:13:32.979 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868690798 ms)
2025-02-17 21:13:32.979 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868690799 ms)
2025-02-17 21:13:32.980 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868690799 ms)
2025-02-17 21:13:32.980 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868690800 ms)
2025-02-17 21:13:32.981 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868690800 ms)
2025-02-17 21:13:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868693433 ms)
2025-02-17 21:13:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868693434 ms)
2025-02-17 21:13:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:13:25.400202+00:00'} (T+868693435 ms)
2025-02-17 21:14:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868744410 ms)
2025-02-17 21:14:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868744410 ms)
2025-02-17 21:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:14:25.402808+00:00', 'runtime': '868740.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.435', 'adc_il_f': '0.372', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.549', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868744414 ms)
2025-02-17 21:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:13:25.402837+00:00', 'runtime': '868680.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.131', 'adc_vl_f': '14.34', 'adc_ic_f': '1.437', 'adc_il_f': '0.338', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.817', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868744414 ms)
2025-02-17 21:14:35.633 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868753452 ms)
2025-02-17 21:14:35.633 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868753453 ms)
2025-02-17 21:14:35.634 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:14:25.400202+00:00'} (T+868753454 ms)
2025-02-17 21:14:45.163 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+868762982 ms)
2025-02-17 21:14:45.163 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+868762982 ms)
2025-02-17 21:14:45.163 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34830)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868762983 ms)
2025-02-17 21:15:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868804411 ms)
2025-02-17 21:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868804411 ms)
2025-02-17 21:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:15:25.402806+00:00', 'runtime': '868800.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.208', 'adc_vl_f': '14.34', 'adc_ic_f': '1.437', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.549', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868804415 ms)
2025-02-17 21:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:14:25.402808+00:00', 'runtime': '868740.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.435', 'adc_il_f': '0.372', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.549', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868804416 ms)
2025-02-17 21:15:45.284 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+868823104 ms)
2025-02-17 21:15:45.285 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+868823104 ms)
2025-02-17 21:15:45.285 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34902)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868823104 ms)
2025-02-17 21:16:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868864413 ms)
2025-02-17 21:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868864413 ms)
2025-02-17 21:16:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:16:25.400830+00:00', 'runtime': '868860.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.452', 'adc_il_f': '0.343', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.791', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868864417 ms)
2025-02-17 21:16:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:15:25.402806+00:00', 'runtime': '868800.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.208', 'adc_vl_f': '14.34', 'adc_ic_f': '1.437', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.8', 'ahc_t': '2657.8', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.549', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.8', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868864417 ms)
2025-02-17 21:17:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868924413 ms)
2025-02-17 21:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868924413 ms)
2025-02-17 21:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:17:25.402831+00:00', 'runtime': '868920.003', 'adc_vb_f': '14.352', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.459', 'adc_il_f': '0.324', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.927', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868924415 ms)
2025-02-17 21:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:16:25.400830+00:00', 'runtime': '868860.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.223', 'adc_vl_f': '14.34', 'adc_ic_f': '1.452', 'adc_il_f': '0.343', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.791', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868924416 ms)
2025-02-17 21:17:35.794 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868933614 ms)
2025-02-17 21:17:35.795 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+868933614 ms)
2025-02-17 21:17:35.796 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:17:25.400189+00:00'} (T+868933615 ms)
2025-02-17 21:17:52.545 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+868950364 ms)
2025-02-17 21:17:52.545 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+868950364 ms)
2025-02-17 21:17:52.545 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35040)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868950365 ms)
2025-02-17 21:18:18.576 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868976395 ms)
2025-02-17 21:18:18.576 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868976396 ms)
2025-02-17 21:18:18.577 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868976396 ms)
2025-02-17 21:18:18.578 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+868976397 ms)
2025-02-17 21:18:18.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+868976397 ms)
2025-02-17 21:18:18.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+868976397 ms)
2025-02-17 21:18:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+868984417 ms)
2025-02-17 21:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+868984418 ms)
2025-02-17 21:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:18:25.402631+00:00', 'runtime': '868980.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.192', 'adc_vl_f': '14.34', 'adc_ic_f': '1.454', 'adc_il_f': '0.331', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.791', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868984419 ms)
2025-02-17 21:18:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:17:25.402831+00:00', 'runtime': '868920.003', 'adc_vb_f': '14.352', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.459', 'adc_il_f': '0.324', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.927', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+868984420 ms)
2025-02-17 21:19:04.638 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+869022457 ms)
2025-02-17 21:19:04.638 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+869022457 ms)
2025-02-17 21:19:04.638 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869022458 ms)
2025-02-17 21:19:10.715 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+869028535 ms)
2025-02-17 21:19:10.716 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+869028535 ms)
2025-02-17 21:19:10.716 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35126)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869028535 ms)
2025-02-17 21:19:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869044402 ms)
2025-02-17 21:19:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869044402 ms)
2025-02-17 21:19:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:19:25.400804+00:00', 'runtime': '869040.001', 'adc_vb_f': '14.359', 'adc_va_f': '22.006', 'adc_vl_f': '14.34', 'adc_ic_f': '1.587', 'adc_il_f': '0.304', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.455', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869044404 ms)
2025-02-17 21:19:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:18:25.402631+00:00', 'runtime': '868980.003', 'adc_vb_f': '14.346', 'adc_va_f': '22.192', 'adc_vl_f': '14.34', 'adc_ic_f': '1.454', 'adc_il_f': '0.331', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.791', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869044404 ms)
2025-02-17 21:19:52.769 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+869070588 ms)
2025-02-17 21:19:52.769 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+869070588 ms)
2025-02-17 21:19:52.770 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869070589 ms)
2025-02-17 21:19:52.770 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+869070590 ms)
2025-02-17 21:19:52.771 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+869070590 ms)
2025-02-17 21:19:52.771 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869070590 ms)
2025-02-17 21:20:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869104403 ms)
2025-02-17 21:20:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869104404 ms)
2025-02-17 21:20:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:20:25.400188+00:00', 'runtime': '869100.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.307', 'adc_ic_f': '1.442', 'adc_il_f': '0.341', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.331', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.229', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869104405 ms)
2025-02-17 21:20:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:19:25.400804+00:00', 'runtime': '869040.001', 'adc_vb_f': '14.359', 'adc_va_f': '22.006', 'adc_vl_f': '14.34', 'adc_ic_f': '1.587', 'adc_il_f': '0.304', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2657.9', 'ahc_t': '2657.9', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.455', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '27.9', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869104405 ms)
2025-02-17 21:21:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869164408 ms)
2025-02-17 21:21:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869164408 ms)
2025-02-17 21:21:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:21:25.400834+00:00', 'runtime': '869160.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.304', 'adc_ic_f': '1.459', 'adc_il_f': '0.338', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869164412 ms)
2025-02-17 21:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:20:25.400188+00:00', 'runtime': '869100.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.307', 'adc_ic_f': '1.442', 'adc_il_f': '0.341', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.331', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.229', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869164413 ms)
2025-02-17 21:21:29.979 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+869167798 ms)
2025-02-17 21:21:29.979 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+869167798 ms)
2025-02-17 21:21:29.980 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35272)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869167799 ms)
2025-02-17 21:22:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869224413 ms)
2025-02-17 21:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869224414 ms)
2025-02-17 21:22:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:22:25.400589+00:00', 'runtime': '869220.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.092', 'adc_vl_f': '14.304', 'adc_ic_f': '1.474', 'adc_il_f': '0.367', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.972', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869224415 ms)
2025-02-17 21:22:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:21:25.400834+00:00', 'runtime': '869160.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.304', 'adc_ic_f': '1.459', 'adc_il_f': '0.338', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869224416 ms)
2025-02-17 21:23:26.121 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+869283941 ms)
2025-02-17 21:23:26.122 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+869283941 ms)
2025-02-17 21:23:26.122 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869283941 ms)
2025-02-17 21:23:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869284415 ms)
2025-02-17 21:23:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869284415 ms)
2025-02-17 21:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:23:25.403091+00:00', 'runtime': '869280.004', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.301', 'adc_ic_f': '1.481', 'adc_il_f': '0.37', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.168', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869284417 ms)
2025-02-17 21:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:22:25.400589+00:00', 'runtime': '869220.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.092', 'adc_vl_f': '14.304', 'adc_ic_f': '1.474', 'adc_il_f': '0.367', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.972', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869284417 ms)
2025-02-17 21:24:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869344406 ms)
2025-02-17 21:24:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869344406 ms)
2025-02-17 21:24:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:24:25.400011+00:00', 'runtime': '869340.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.095', 'adc_vl_f': '14.304', 'adc_ic_f': '1.462', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.032', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869344408 ms)
2025-02-17 21:24:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:23:25.403091+00:00', 'runtime': '869280.004', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.301', 'adc_ic_f': '1.481', 'adc_il_f': '0.37', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.0', 'ahc_t': '2658.0', 'kwhc': '34.4', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.168', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.0', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869344409 ms)
2025-02-17 21:25:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869404407 ms)
2025-02-17 21:25:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869404408 ms)
2025-02-17 21:25:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:25:25.400190+00:00', 'runtime': '869400.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.064', 'adc_vl_f': '14.304', 'adc_ic_f': '1.454', 'adc_il_f': '0.358', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.927', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869404409 ms)
2025-02-17 21:25:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:24:25.400011+00:00', 'runtime': '869340.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.095', 'adc_vl_f': '14.304', 'adc_ic_f': '1.462', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.032', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869404410 ms)
2025-02-17 21:25:35.411 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+869413230 ms)
2025-02-17 21:25:35.412 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+869413231 ms)
2025-02-17 21:25:35.412 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:25:25.405063+00:00'} (T+869413232 ms)
2025-02-17 21:26:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869464410 ms)
2025-02-17 21:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869464410 ms)
2025-02-17 21:26:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:26:25.402817+00:00', 'runtime': '869460.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.092', 'adc_vl_f': '14.304', 'adc_ic_f': '1.435', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.41', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869464411 ms)
2025-02-17 21:26:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:25:25.400190+00:00', 'runtime': '869400.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.064', 'adc_vl_f': '14.304', 'adc_ic_f': '1.454', 'adc_il_f': '0.358', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.927', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869464412 ms)
2025-02-17 21:26:35.682 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+869473501 ms)
2025-02-17 21:26:35.682 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+869473501 ms)
2025-02-17 21:26:35.683 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:26:25.400218+00:00'} (T+869473502 ms)
2025-02-17 21:27:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869524414 ms)
2025-02-17 21:27:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869524414 ms)
2025-02-17 21:27:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:27:25.402722+00:00', 'runtime': '869520.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.095', 'adc_vl_f': '14.304', 'adc_ic_f': '1.382', 'adc_il_f': '0.319', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.987', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869524419 ms)
2025-02-17 21:27:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:26:25.402817+00:00', 'runtime': '869460.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.092', 'adc_vl_f': '14.304', 'adc_ic_f': '1.435', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.41', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869524419 ms)
2025-02-17 21:27:44.445 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+869542264 ms)
2025-02-17 21:27:44.446 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+869542265 ms)
2025-02-17 21:27:44.446 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869542265 ms)
2025-02-17 21:27:44.447 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+869542266 ms)
2025-02-17 21:27:44.447 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+869542266 ms)
2025-02-17 21:27:44.447 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869542266 ms)
2025-02-17 21:28:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869584418 ms)
2025-02-17 21:28:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869584419 ms)
2025-02-17 21:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:28:25.399988+00:00', 'runtime': '869580.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.028', 'adc_vl_f': '14.304', 'adc_ic_f': '1.423', 'adc_il_f': '0.316', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.991', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869584423 ms)
2025-02-17 21:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:27:25.402722+00:00', 'runtime': '869520.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.095', 'adc_vl_f': '14.304', 'adc_ic_f': '1.382', 'adc_il_f': '0.319', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.1', 'ahc_t': '2658.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.987', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.1', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869584423 ms)
2025-02-17 21:28:38.584 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+869596404 ms)
2025-02-17 21:28:38.585 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+869596404 ms)
2025-02-17 21:28:38.585 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35734)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869596404 ms)
2025-02-17 21:28:53.694 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+869611513 ms)
2025-02-17 21:28:53.694 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+869611514 ms)
2025-02-17 21:28:53.695 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35752)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869611514 ms)
2025-02-17 21:29:12.783 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+869630602 ms)
2025-02-17 21:29:12.783 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+869630603 ms)
2025-02-17 21:29:12.784 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35772)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869630603 ms)
2025-02-17 21:29:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869644419 ms)
2025-02-17 21:29:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869644419 ms)
2025-02-17 21:29:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:29:25.400164+00:00', 'runtime': '869640.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.307', 'adc_ic_f': '1.396', 'adc_il_f': '0.324', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869644423 ms)
2025-02-17 21:29:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:28:25.399988+00:00', 'runtime': '869580.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.028', 'adc_vl_f': '14.304', 'adc_ic_f': '1.423', 'adc_il_f': '0.316', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.991', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869644423 ms)
2025-02-17 21:29:35.653 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+869653472 ms)
2025-02-17 21:29:35.653 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+869653473 ms)
2025-02-17 21:29:35.654 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:29:25.413049+00:00'} (T+869653474 ms)
2025-02-17 21:30:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869704419 ms)
2025-02-17 21:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869704419 ms)
2025-02-17 21:30:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:30:25.402870+00:00', 'runtime': '869700.003', 'adc_vb_f': '14.325', 'adc_va_f': '22.122', 'adc_vl_f': '14.304', 'adc_ic_f': '1.466', 'adc_il_f': '0.367', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869704424 ms)
2025-02-17 21:30:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:29:25.400164+00:00', 'runtime': '869640.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.095', 'adc_vl_f': '14.307', 'adc_ic_f': '1.396', 'adc_il_f': '0.324', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869704424 ms)
2025-02-17 21:31:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869764406 ms)
2025-02-17 21:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869764407 ms)
2025-02-17 21:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:31:25.400590+00:00', 'runtime': '869760.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.269', 'adc_vl_f': '14.31', 'adc_ic_f': '1.447', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.232', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869764411 ms)
2025-02-17 21:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:30:25.402870+00:00', 'runtime': '869700.003', 'adc_vb_f': '14.325', 'adc_va_f': '22.122', 'adc_vl_f': '14.304', 'adc_ic_f': '1.466', 'adc_il_f': '0.367', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.3', 'ahl_t': '1592.3', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869764411 ms)
2025-02-17 21:32:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869824409 ms)
2025-02-17 21:32:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869824409 ms)
2025-02-17 21:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:32:25.400606+00:00', 'runtime': '869820.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.214', 'adc_vl_f': '14.304', 'adc_ic_f': '1.447', 'adc_il_f': '0.36', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.761', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869824413 ms)
2025-02-17 21:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:31:25.400590+00:00', 'runtime': '869760.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.269', 'adc_vl_f': '14.31', 'adc_ic_f': '1.447', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.2', 'ahc_t': '2658.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.232', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.2', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869824414 ms)
2025-02-17 21:33:17.159 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+869874979 ms)
2025-02-17 21:33:17.160 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+869874979 ms)
2025-02-17 21:33:17.160 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36030)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+869874979 ms)
2025-02-17 21:33:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869884412 ms)
2025-02-17 21:33:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869884413 ms)
2025-02-17 21:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:33:25.400188+00:00', 'runtime': '869880.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.305', 'adc_vl_f': '14.307', 'adc_ic_f': '1.435', 'adc_il_f': '0.36', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.002', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869884416 ms)
2025-02-17 21:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:32:25.400606+00:00', 'runtime': '869820.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.214', 'adc_vl_f': '14.304', 'adc_ic_f': '1.447', 'adc_il_f': '0.36', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.761', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869884416 ms)
2025-02-17 21:34:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+869944415 ms)
2025-02-17 21:34:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+869944415 ms)
2025-02-17 21:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:34:25.400203+00:00', 'runtime': '869940.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.253', 'adc_vl_f': '14.301', 'adc_ic_f': '1.454', 'adc_il_f': '0.36', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.851', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869944416 ms)
2025-02-17 21:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:33:25.400188+00:00', 'runtime': '869880.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.305', 'adc_vl_f': '14.307', 'adc_ic_f': '1.435', 'adc_il_f': '0.36', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.002', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+869944417 ms)
2025-02-17 21:35:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870004402 ms)
2025-02-17 21:35:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870004403 ms)
2025-02-17 21:35:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:35:25.400198+00:00', 'runtime': '870000.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.211', 'adc_vl_f': '14.307', 'adc_ic_f': '1.433', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870004404 ms)
2025-02-17 21:35:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:34:25.400203+00:00', 'runtime': '869940.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.253', 'adc_vl_f': '14.301', 'adc_ic_f': '1.454', 'adc_il_f': '0.36', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.851', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870004404 ms)
2025-02-17 21:36:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870064405 ms)
2025-02-17 21:36:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870064406 ms)
2025-02-17 21:36:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:36:25.400203+00:00', 'runtime': '870060.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.293', 'adc_vl_f': '14.304', 'adc_ic_f': '1.408', 'adc_il_f': '0.343', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870064407 ms)
2025-02-17 21:36:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:35:25.400198+00:00', 'runtime': '870000.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.211', 'adc_vl_f': '14.307', 'adc_ic_f': '1.433', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.3', 'ahc_t': '2658.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.896', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.3', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870064407 ms)
2025-02-17 21:36:35.661 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870073481 ms)
2025-02-17 21:36:35.662 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870073481 ms)
2025-02-17 21:36:35.663 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:36:25.400334+00:00'} (T+870073482 ms)
2025-02-17 21:37:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870124407 ms)
2025-02-17 21:37:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870124407 ms)
2025-02-17 21:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:37:25.400194+00:00', 'runtime': '870120.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.226', 'adc_vl_f': '14.304', 'adc_ic_f': '1.396', 'adc_il_f': '0.321', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.957', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870124411 ms)
2025-02-17 21:37:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:36:25.400203+00:00', 'runtime': '870060.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.293', 'adc_vl_f': '14.304', 'adc_ic_f': '1.408', 'adc_il_f': '0.343', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.138', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870124412 ms)
2025-02-17 21:37:36.151 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870133970 ms)
2025-02-17 21:37:36.151 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870133970 ms)
2025-02-17 21:37:36.152 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:37:25.400144+00:00'} (T+870133971 ms)
2025-02-17 21:37:46.589 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+870144408 ms)
2025-02-17 21:37:46.589 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+870144408 ms)
2025-02-17 21:37:46.589 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36324)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+870144409 ms)
2025-02-17 21:38:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870184411 ms)
2025-02-17 21:38:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870184411 ms)
2025-02-17 21:38:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:38:25.400168+00:00', 'runtime': '870180.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.205', 'adc_vl_f': '14.304', 'adc_ic_f': '1.389', 'adc_il_f': '0.329', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.761', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870184415 ms)
2025-02-17 21:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:37:25.400194+00:00', 'runtime': '870120.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.226', 'adc_vl_f': '14.304', 'adc_ic_f': '1.396', 'adc_il_f': '0.321', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.957', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870184416 ms)
2025-02-17 21:38:35.763 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870193582 ms)
2025-02-17 21:38:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870193583 ms)
2025-02-17 21:38:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:38:25.400142+00:00'} (T+870193583 ms)
2025-02-17 21:38:51.708 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+870209527 ms)
2025-02-17 21:38:51.708 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+870209528 ms)
2025-02-17 21:38:51.709 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36396)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+870209528 ms)
2025-02-17 21:39:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870244416 ms)
2025-02-17 21:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870244416 ms)
2025-02-17 21:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:39:25.400174+00:00', 'runtime': '870240.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.416', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.791', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870244417 ms)
2025-02-17 21:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:38:25.400168+00:00', 'runtime': '870180.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.205', 'adc_vl_f': '14.304', 'adc_ic_f': '1.389', 'adc_il_f': '0.329', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.761', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870244417 ms)
2025-02-17 21:40:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870304404 ms)
2025-02-17 21:40:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870304405 ms)
2025-02-17 21:40:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:40:25.400156+00:00', 'runtime': '870300.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.208', 'adc_vl_f': '14.307', 'adc_ic_f': '1.377', 'adc_il_f': '0.309', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.583', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870304406 ms)
2025-02-17 21:40:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:39:25.400174+00:00', 'runtime': '870240.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.416', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.791', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870304406 ms)
2025-02-17 21:41:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870364408 ms)
2025-02-17 21:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870364408 ms)
2025-02-17 21:41:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:41:25.400184+00:00', 'runtime': '870360.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.202', 'adc_vl_f': '14.301', 'adc_ic_f': '1.435', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.579', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870364409 ms)
2025-02-17 21:41:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:40:25.400156+00:00', 'runtime': '870300.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.208', 'adc_vl_f': '14.307', 'adc_ic_f': '1.377', 'adc_il_f': '0.309', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.4', 'ahc_t': '2658.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.583', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.4', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870364410 ms)
2025-02-17 21:42:13.061 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+870410880 ms)
2025-02-17 21:42:13.061 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+870410881 ms)
2025-02-17 21:42:13.062 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36608)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+870410881 ms)
2025-02-17 21:42:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870424409 ms)
2025-02-17 21:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870424410 ms)
2025-02-17 21:42:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:42:25.400206+00:00', 'runtime': '870420.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.171', 'adc_vl_f': '14.31', 'adc_ic_f': '1.43', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.93', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870424411 ms)
2025-02-17 21:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:41:25.400184+00:00', 'runtime': '870360.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.202', 'adc_vl_f': '14.301', 'adc_ic_f': '1.435', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.579', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870424411 ms)
2025-02-17 21:43:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870484413 ms)
2025-02-17 21:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870484413 ms)
2025-02-17 21:43:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:43:25.400702+00:00', 'runtime': '870480.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.192', 'adc_vl_f': '14.304', 'adc_ic_f': '1.428', 'adc_il_f': '0.348', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.579', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870484417 ms)
2025-02-17 21:43:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:42:25.400206+00:00', 'runtime': '870420.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.171', 'adc_vl_f': '14.31', 'adc_ic_f': '1.43', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.93', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870484418 ms)
2025-02-17 21:43:27.224 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+870485044 ms)
2025-02-17 21:43:27.225 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+870485044 ms)
2025-02-17 21:43:27.225 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36686)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+870485044 ms)
2025-02-17 21:44:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870544416 ms)
2025-02-17 21:44:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870544417 ms)
2025-02-17 21:44:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:44:25.402835+00:00', 'runtime': '870540.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.119', 'adc_vl_f': '14.307', 'adc_ic_f': '1.423', 'adc_il_f': '0.37', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.549', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870544421 ms)
2025-02-17 21:44:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:43:25.400702+00:00', 'runtime': '870480.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.192', 'adc_vl_f': '14.304', 'adc_ic_f': '1.428', 'adc_il_f': '0.348', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.579', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870544421 ms)
2025-02-17 21:44:35.675 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870553495 ms)
2025-02-17 21:44:35.676 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870553495 ms)
2025-02-17 21:44:35.677 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:44:25.400195+00:00'} (T+870553496 ms)
2025-02-17 21:45:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870604421 ms)
2025-02-17 21:45:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870604421 ms)
2025-02-17 21:45:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:45:25.402766+00:00', 'runtime': '870600.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.205', 'adc_vl_f': '14.304', 'adc_ic_f': '1.394', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.519', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870604425 ms)
2025-02-17 21:45:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:44:25.402835+00:00', 'runtime': '870540.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.119', 'adc_vl_f': '14.307', 'adc_ic_f': '1.423', 'adc_il_f': '0.37', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.5', 'ahc_t': '2658.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.549', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.5', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870604426 ms)
2025-02-17 21:45:35.773 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870613592 ms)
2025-02-17 21:45:35.774 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870613593 ms)
2025-02-17 21:45:35.775 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:45:25.400149+00:00'} (T+870613594 ms)
2025-02-17 21:46:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870664407 ms)
2025-02-17 21:46:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870664408 ms)
2025-02-17 21:46:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:46:25.400023+00:00', 'runtime': '870660.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.205', 'adc_vl_f': '14.307', 'adc_ic_f': '1.406', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.821', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870664410 ms)
2025-02-17 21:46:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:45:25.402766+00:00', 'runtime': '870600.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.205', 'adc_vl_f': '14.304', 'adc_ic_f': '1.394', 'adc_il_f': '0.35', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.519', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870664410 ms)
2025-02-17 21:46:47.473 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+870685293 ms)
2025-02-17 21:46:47.474 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+870685293 ms)
2025-02-17 21:46:47.474 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+870685293 ms)
2025-02-17 21:47:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870724413 ms)
2025-02-17 21:47:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870724413 ms)
2025-02-17 21:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:47:25.400663+00:00', 'runtime': '870720.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.287', 'adc_vl_f': '14.304', 'adc_ic_f': '1.341', 'adc_il_f': '0.3', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.477', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870724415 ms)
2025-02-17 21:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:46:25.400023+00:00', 'runtime': '870660.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.205', 'adc_vl_f': '14.307', 'adc_ic_f': '1.406', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.821', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870724415 ms)
2025-02-17 21:48:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870784416 ms)
2025-02-17 21:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870784416 ms)
2025-02-17 21:48:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:48:25.400616+00:00', 'runtime': '870780.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.198', 'adc_vl_f': '14.307', 'adc_ic_f': '1.384', 'adc_il_f': '0.321', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.568', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870784418 ms)
2025-02-17 21:48:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:47:25.400663+00:00', 'runtime': '870720.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.287', 'adc_vl_f': '14.304', 'adc_ic_f': '1.341', 'adc_il_f': '0.3', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.477', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870784419 ms)
2025-02-17 21:49:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870844417 ms)
2025-02-17 21:49:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870844418 ms)
2025-02-17 21:49:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:49:25.402791+00:00', 'runtime': '870840.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.362', 'adc_il_f': '0.326', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.368', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870844420 ms)
2025-02-17 21:49:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:48:25.400616+00:00', 'runtime': '870780.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.198', 'adc_vl_f': '14.307', 'adc_ic_f': '1.384', 'adc_il_f': '0.321', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.6', 'ahc_t': '2658.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.4', 'ahl_t': '1592.4', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.568', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.6', 'ahl_daily': '8.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870844420 ms)
2025-02-17 21:49:35.849 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870853668 ms)
2025-02-17 21:49:35.850 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870853669 ms)
2025-02-17 21:49:35.851 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:49:25.400142+00:00'} (T+870853670 ms)
2025-02-17 21:50:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870904407 ms)
2025-02-17 21:50:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870904408 ms)
2025-02-17 21:50:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:50:25.400170+00:00', 'runtime': '870900.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.11', 'adc_vl_f': '14.307', 'adc_ic_f': '1.394', 'adc_il_f': '0.341', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870904412 ms)
2025-02-17 21:50:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:49:25.402791+00:00', 'runtime': '870840.003', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.362', 'adc_il_f': '0.326', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.368', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870904412 ms)
2025-02-17 21:50:35.734 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870913553 ms)
2025-02-17 21:50:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870913553 ms)
2025-02-17 21:50:35.735 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:50:25.400472+00:00'} (T+870913554 ms)
2025-02-17 21:51:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+870964409 ms)
2025-02-17 21:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+870964410 ms)
2025-02-17 21:51:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:51:25.400191+00:00', 'runtime': '870960.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.177', 'adc_vl_f': '14.31', 'adc_ic_f': '1.362', 'adc_il_f': '0.331', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.398', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870964414 ms)
2025-02-17 21:51:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:50:25.400170+00:00', 'runtime': '870900.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.11', 'adc_vl_f': '14.307', 'adc_ic_f': '1.394', 'adc_il_f': '0.341', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+870964414 ms)
2025-02-17 21:51:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870973422 ms)
2025-02-17 21:51:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+870973422 ms)
2025-02-17 21:51:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:51:25.400060+00:00'} (T+870973424 ms)
2025-02-17 21:51:51.884 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+870989704 ms)
2025-02-17 21:51:51.885 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+870989704 ms)
2025-02-17 21:51:51.885 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37232)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+870989704 ms)
2025-02-17 21:52:19.025 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871016845 ms)
2025-02-17 21:52:19.026 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871016845 ms)
2025-02-17 21:52:19.026 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37260)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871016845 ms)
2025-02-17 21:52:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871024412 ms)
2025-02-17 21:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871024413 ms)
2025-02-17 21:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:52:25.400176+00:00', 'runtime': '871020.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.104', 'adc_vl_f': '14.304', 'adc_ic_f': '1.411', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.338', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871024416 ms)
2025-02-17 21:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:51:25.400191+00:00', 'runtime': '870960.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.177', 'adc_vl_f': '14.31', 'adc_ic_f': '1.362', 'adc_il_f': '0.331', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.398', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871024416 ms)
2025-02-17 21:52:37.039 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871034859 ms)
2025-02-17 21:52:37.040 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871034859 ms)
2025-02-17 21:52:37.041 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:52:25.400203+00:00'} (T+871034860 ms)
2025-02-17 21:53:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871084416 ms)
2025-02-17 21:53:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871084417 ms)
2025-02-17 21:53:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:53:25.400156+00:00', 'runtime': '871080.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.183', 'adc_vl_f': '14.307', 'adc_ic_f': '1.408', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.232', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871084421 ms)
2025-02-17 21:53:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:52:25.400176+00:00', 'runtime': '871020.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.104', 'adc_vl_f': '14.304', 'adc_ic_f': '1.411', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.7', 'ahc_t': '2658.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.338', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.7', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871084421 ms)
2025-02-17 21:53:36.063 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871093882 ms)
2025-02-17 21:53:36.064 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871093883 ms)
2025-02-17 21:53:36.065 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:53:25.400273+00:00'} (T+871093884 ms)
2025-02-17 21:54:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871144403 ms)
2025-02-17 21:54:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871144404 ms)
2025-02-17 21:54:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:54:25.400181+00:00', 'runtime': '871140.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.186', 'adc_vl_f': '14.301', 'adc_ic_f': '1.411', 'adc_il_f': '0.37', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.172', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871144405 ms)
2025-02-17 21:54:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:53:25.400156+00:00', 'runtime': '871080.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.183', 'adc_vl_f': '14.307', 'adc_ic_f': '1.408', 'adc_il_f': '0.362', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20128', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.232', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871144405 ms)
2025-02-17 21:54:36.326 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871154145 ms)
2025-02-17 21:54:36.326 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871154146 ms)
2025-02-17 21:54:36.327 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37408)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871154146 ms)
2025-02-17 21:55:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871204407 ms)
2025-02-17 21:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871204407 ms)
2025-02-17 21:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:55:25.400049+00:00', 'runtime': '871200.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.307', 'adc_ic_f': '1.355', 'adc_il_f': '0.324', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.368', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871204409 ms)
2025-02-17 21:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:54:25.400181+00:00', 'runtime': '871140.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.186', 'adc_vl_f': '14.301', 'adc_ic_f': '1.411', 'adc_il_f': '0.37', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.172', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871204409 ms)
2025-02-17 21:56:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871264410 ms)
2025-02-17 21:56:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871264410 ms)
2025-02-17 21:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:56:25.400184+00:00', 'runtime': '871260.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.198', 'adc_vl_f': '14.304', 'adc_ic_f': '1.362', 'adc_il_f': '0.333', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.157', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871264414 ms)
2025-02-17 21:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:55:25.400049+00:00', 'runtime': '871200.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.307', 'adc_ic_f': '1.355', 'adc_il_f': '0.324', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.368', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871264414 ms)
2025-02-17 21:56:35.748 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871273567 ms)
2025-02-17 21:56:35.749 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871273568 ms)
2025-02-17 21:56:35.750 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:56:25.400304+00:00'} (T+871273569 ms)
2025-02-17 21:57:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871324412 ms)
2025-02-17 21:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871324412 ms)
2025-02-17 21:57:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:57:25.400192+00:00', 'runtime': '871320.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.307', 'adc_ic_f': '1.37', 'adc_il_f': '0.341', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.368', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871324417 ms)
2025-02-17 21:57:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:56:25.400184+00:00', 'runtime': '871260.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.198', 'adc_vl_f': '14.304', 'adc_ic_f': '1.362', 'adc_il_f': '0.333', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.157', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871324418 ms)
2025-02-17 21:57:36.053 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871333873 ms)
2025-02-17 21:57:36.054 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871333873 ms)
2025-02-17 21:57:36.055 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:57:25.400065+00:00'} (T+871333874 ms)
2025-02-17 21:57:59.643 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871357462 ms)
2025-02-17 21:57:59.643 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871357462 ms)
2025-02-17 21:57:59.643 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37634)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871357462 ms)
2025-02-17 21:58:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871384415 ms)
2025-02-17 21:58:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871384416 ms)
2025-02-17 21:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:58:25.400539+00:00', 'runtime': '871380.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.205', 'adc_vl_f': '14.301', 'adc_ic_f': '1.353', 'adc_il_f': '0.336', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871384419 ms)
2025-02-17 21:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:57:25.400192+00:00', 'runtime': '871320.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.307', 'adc_ic_f': '1.37', 'adc_il_f': '0.341', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.8', 'ahc_t': '2658.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.368', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.8', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871384420 ms)
2025-02-17 21:58:27.753 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871385573 ms)
2025-02-17 21:58:27.754 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871385573 ms)
2025-02-17 21:58:27.754 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37664)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871385573 ms)
2025-02-17 21:58:35.745 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871393564 ms)
2025-02-17 21:58:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871393565 ms)
2025-02-17 21:58:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:58:25.400262+00:00'} (T+871393566 ms)
2025-02-17 21:59:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871444406 ms)
2025-02-17 21:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871444406 ms)
2025-02-17 21:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 21:59:25.400167+00:00', 'runtime': '871440.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.365', 'adc_il_f': '0.333', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871444407 ms)
2025-02-17 21:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:58:25.400539+00:00', 'runtime': '871380.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.205', 'adc_vl_f': '14.301', 'adc_ic_f': '1.353', 'adc_il_f': '0.336', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871444407 ms)
2025-02-17 21:59:35.681 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871453500 ms)
2025-02-17 21:59:35.681 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871453500 ms)
2025-02-17 21:59:35.682 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 21:59:25.400052+00:00'} (T+871453501 ms)
2025-02-17 22:00:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871504407 ms)
2025-02-17 22:00:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871504407 ms)
2025-02-17 22:00:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:00:25.400167+00:00', 'runtime': '871500.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.362', 'adc_il_f': '0.343', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.096', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871504408 ms)
2025-02-17 22:00:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 21:59:25.400167+00:00', 'runtime': '871440.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.365', 'adc_il_f': '0.333', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.262', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871504409 ms)
2025-02-17 22:00:35.830 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871513649 ms)
2025-02-17 22:00:35.831 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871513650 ms)
2025-02-17 22:00:35.832 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:00:25.400109+00:00'} (T+871513651 ms)
2025-02-17 22:01:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871564410 ms)
2025-02-17 22:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871564411 ms)
2025-02-17 22:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:01:25.400179+00:00', 'runtime': '871560.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.396', 'adc_il_f': '0.353', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.021', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871564412 ms)
2025-02-17 22:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:00:25.400167+00:00', 'runtime': '871500.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.362', 'adc_il_f': '0.343', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.096', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871564412 ms)
2025-02-17 22:01:35.761 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871573580 ms)
2025-02-17 22:01:35.762 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871573581 ms)
2025-02-17 22:01:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:01:25.399998+00:00'} (T+871573582 ms)
2025-02-17 22:02:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871624412 ms)
2025-02-17 22:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871624412 ms)
2025-02-17 22:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:02:25.400179+00:00', 'runtime': '871620.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.307', 'adc_ic_f': '1.391', 'adc_il_f': '0.372', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.096', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871624413 ms)
2025-02-17 22:02:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:01:25.400179+00:00', 'runtime': '871560.001', 'adc_vb_f': '14.325', 'adc_va_f': '22.202', 'adc_vl_f': '14.304', 'adc_ic_f': '1.396', 'adc_il_f': '0.353', 't_hs': '23', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.853', 'ahc_r': '2658.9', 'ahc_t': '2658.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.021', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '28.9', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871624414 ms)
2025-02-17 22:02:38.123 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871635943 ms)
2025-02-17 22:02:38.124 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871635943 ms)
2025-02-17 22:02:38.124 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37934)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871635943 ms)
2025-02-17 22:02:40.183 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871638002 ms)
2025-02-17 22:02:40.183 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871638002 ms)
2025-02-17 22:02:40.183 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37938)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871638003 ms)
2025-02-17 22:02:57.282 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871655101 ms)
2025-02-17 22:02:57.282 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871655102 ms)
2025-02-17 22:02:57.283 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37958)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871655102 ms)
2025-02-17 22:03:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871684411 ms)
2025-02-17 22:03:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871684412 ms)
2025-02-17 22:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:03:25.400194+00:00', 'runtime': '871680.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.35', 'adc_il_f': '0.348', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.34', 'vb_ref': '13.883', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.341', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871684415 ms)
2025-02-17 22:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:02:25.400179+00:00', 'runtime': '871620.001', 'adc_vb_f': '14.322', 'adc_va_f': '22.202', 'adc_vl_f': '14.307', 'adc_ic_f': '1.391', 'adc_il_f': '0.372', 't_hs': '22', 't_batt': '24', 't_amb': '24', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.322', 'vb_ref': '13.853', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.096', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871684416 ms)
2025-02-17 22:03:51.391 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871709210 ms)
2025-02-17 22:03:51.391 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871709210 ms)
2025-02-17 22:03:51.391 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38016)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871709211 ms)
2025-02-17 22:04:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871744411 ms)
2025-02-17 22:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871744412 ms)
2025-02-17 22:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:04:25.400157+00:00', 'runtime': '871740.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.341', 'adc_il_f': '0.326', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.266', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871744416 ms)
2025-02-17 22:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:03:25.400194+00:00', 'runtime': '871680.001', 'adc_vb_f': '14.349', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.35', 'adc_il_f': '0.348', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.34', 'vb_ref': '13.883', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.341', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871744416 ms)
2025-02-17 22:04:36.958 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871754777 ms)
2025-02-17 22:04:36.959 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871754778 ms)
2025-02-17 22:04:36.960 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:04:25.400337+00:00'} (T+871754779 ms)
2025-02-17 22:05:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871804411 ms)
2025-02-17 22:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871804412 ms)
2025-02-17 22:05:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:05:25.400488+00:00', 'runtime': '871800.001', 'adc_vb_f': '14.355', 'adc_va_f': '22.186', 'adc_vl_f': '14.34', 'adc_ic_f': '1.384', 'adc_il_f': '0.326', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.341', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871804413 ms)
2025-02-17 22:05:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:04:25.400157+00:00', 'runtime': '871740.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.341', 'adc_il_f': '0.326', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.266', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871804414 ms)
2025-02-17 22:05:35.886 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871813705 ms)
2025-02-17 22:05:35.886 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871813706 ms)
2025-02-17 22:05:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:05:25.400377+00:00'} (T+871813706 ms)
2025-02-17 22:06:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871864414 ms)
2025-02-17 22:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871864414 ms)
2025-02-17 22:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:06:25.400152+00:00', 'runtime': '871860.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.331', 'adc_il_f': '0.331', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.085', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871864417 ms)
2025-02-17 22:06:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:05:25.400488+00:00', 'runtime': '871800.001', 'adc_vb_f': '14.355', 'adc_va_f': '22.186', 'adc_vl_f': '14.34', 'adc_ic_f': '1.384', 'adc_il_f': '0.326', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.0', 'ahc_t': '2659.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.341', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.0', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871864418 ms)
2025-02-17 22:06:35.464 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871873284 ms)
2025-02-17 22:06:35.465 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871873284 ms)
2025-02-17 22:06:35.469 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:06:25.400446+00:00'} (T+871873289 ms)
2025-02-17 22:07:09.704 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871907524 ms)
2025-02-17 22:07:09.705 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871907524 ms)
2025-02-17 22:07:09.705 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38238)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871907524 ms)
2025-02-17 22:07:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871924412 ms)
2025-02-17 22:07:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871924412 ms)
2025-02-17 22:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:07:25.400183+00:00', 'runtime': '871920.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.141', 'adc_vl_f': '14.34', 'adc_ic_f': '1.343', 'adc_il_f': '0.321', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.22', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871924414 ms)
2025-02-17 22:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:06:25.400152+00:00', 'runtime': '871860.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.202', 'adc_vl_f': '14.34', 'adc_ic_f': '1.331', 'adc_il_f': '0.331', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.085', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871924415 ms)
2025-02-17 22:07:35.734 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871933553 ms)
2025-02-17 22:07:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871933554 ms)
2025-02-17 22:07:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:07:25.400357+00:00'} (T+871933558 ms)
2025-02-17 22:07:42.821 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+871940640 ms)
2025-02-17 22:07:42.821 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+871940640 ms)
2025-02-17 22:07:42.821 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38274)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+871940641 ms)
2025-02-17 22:08:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+871984414 ms)
2025-02-17 22:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+871984415 ms)
2025-02-17 22:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:08:25.400181+00:00', 'runtime': '871980.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.095', 'adc_vl_f': '14.34', 'adc_ic_f': '1.353', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.402', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871984416 ms)
2025-02-17 22:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:07:25.400183+00:00', 'runtime': '871920.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.141', 'adc_vl_f': '14.34', 'adc_ic_f': '1.343', 'adc_il_f': '0.321', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.5', 'ahl_t': '1592.5', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.22', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+871984417 ms)
2025-02-17 22:08:35.652 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871993471 ms)
2025-02-17 22:08:35.652 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+871993472 ms)
2025-02-17 22:08:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:08:25.400066+00:00'} (T+871993477 ms)
2025-02-17 22:09:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872044414 ms)
2025-02-17 22:09:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872044414 ms)
2025-02-17 22:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:09:25.400208+00:00', 'runtime': '872040.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.055', 'adc_vl_f': '14.34', 'adc_ic_f': '1.319', 'adc_il_f': '0.37', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.843', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872044418 ms)
2025-02-17 22:09:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:08:25.400181+00:00', 'runtime': '871980.001', 'adc_vb_f': '14.352', 'adc_va_f': '22.095', 'adc_vl_f': '14.34', 'adc_ic_f': '1.353', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.402', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872044419 ms)
2025-02-17 22:09:35.741 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872053560 ms)
2025-02-17 22:09:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872053561 ms)
2025-02-17 22:09:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:09:25.400212+00:00'} (T+872053565 ms)
2025-02-17 22:10:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872104417 ms)
2025-02-17 22:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872104417 ms)
2025-02-17 22:10:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:10:25.400172+00:00', 'runtime': '872100.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.991', 'adc_vl_f': '14.34', 'adc_ic_f': '1.341', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.19', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872104421 ms)
2025-02-17 22:10:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:09:25.400208+00:00', 'runtime': '872040.001', 'adc_vb_f': '14.346', 'adc_va_f': '22.055', 'adc_vl_f': '14.34', 'adc_ic_f': '1.319', 'adc_il_f': '0.37', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.843', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872104422 ms)
2025-02-17 22:10:36.161 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872113980 ms)
2025-02-17 22:10:36.161 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872113981 ms)
2025-02-17 22:10:36.163 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:10:25.400055+00:00'} (T+872113982 ms)
2025-02-17 22:11:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872164420 ms)
2025-02-17 22:11:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872164420 ms)
2025-02-17 22:11:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:11:25.400526+00:00', 'runtime': '872160.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.771', 'adc_vl_f': '14.34', 'adc_ic_f': '1.341', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.22', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872164424 ms)
2025-02-17 22:11:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:10:25.400172+00:00', 'runtime': '872100.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.991', 'adc_vl_f': '14.34', 'adc_ic_f': '1.341', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.1', 'ahc_t': '2659.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.19', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.1', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872164425 ms)
2025-02-17 22:11:35.748 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872173567 ms)
2025-02-17 22:11:35.748 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872173568 ms)
2025-02-17 22:11:35.749 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:11:25.400168+00:00'} (T+872173569 ms)
2025-02-17 22:12:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872224420 ms)
2025-02-17 22:12:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872224420 ms)
2025-02-17 22:12:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:12:25.400177+00:00', 'runtime': '872220.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.997', 'adc_vl_f': '14.34', 'adc_ic_f': '1.334', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.085', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872224425 ms)
2025-02-17 22:12:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:11:25.400526+00:00', 'runtime': '872160.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.771', 'adc_vl_f': '14.34', 'adc_ic_f': '1.341', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.22', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872224425 ms)
2025-02-17 22:12:35.679 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872233499 ms)
2025-02-17 22:12:35.680 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872233499 ms)
2025-02-17 22:12:35.682 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:12:25.400039+00:00'} (T+872233501 ms)
2025-02-17 22:13:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872284406 ms)
2025-02-17 22:13:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872284406 ms)
2025-02-17 22:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:13:25.400181+00:00', 'runtime': '872280.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.991', 'adc_vl_f': '14.34', 'adc_ic_f': '1.329', 'adc_il_f': '0.358', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.482', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872284410 ms)
2025-02-17 22:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:12:25.400177+00:00', 'runtime': '872220.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.997', 'adc_vl_f': '14.34', 'adc_ic_f': '1.334', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.085', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872284410 ms)
2025-02-17 22:13:35.759 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872293578 ms)
2025-02-17 22:13:35.760 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872293579 ms)
2025-02-17 22:13:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:13:25.400131+00:00'} (T+872293583 ms)
2025-02-17 22:14:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872344406 ms)
2025-02-17 22:14:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872344407 ms)
2025-02-17 22:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:14:25.400207+00:00', 'runtime': '872340.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.985', 'adc_vl_f': '14.34', 'adc_ic_f': '1.338', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.059', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872344411 ms)
2025-02-17 22:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:13:25.400181+00:00', 'runtime': '872280.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.991', 'adc_vl_f': '14.34', 'adc_ic_f': '1.329', 'adc_il_f': '0.358', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.482', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872344412 ms)
2025-02-17 22:14:35.716 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872353536 ms)
2025-02-17 22:14:35.717 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872353536 ms)
2025-02-17 22:14:35.718 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:14:25.400045+00:00'} (T+872353538 ms)
2025-02-17 22:15:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872404407 ms)
2025-02-17 22:15:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872404407 ms)
2025-02-17 22:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:15:25.400187+00:00', 'runtime': '872400.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.909', 'adc_vl_f': '14.34', 'adc_ic_f': '1.326', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.054', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872404412 ms)
2025-02-17 22:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:14:25.400207+00:00', 'runtime': '872340.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.985', 'adc_vl_f': '14.34', 'adc_ic_f': '1.338', 'adc_il_f': '0.345', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.2', 'ahc_t': '2659.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.059', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.2', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872404412 ms)
2025-02-17 22:16:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872464407 ms)
2025-02-17 22:16:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872464408 ms)
2025-02-17 22:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:16:25.400199+00:00', 'runtime': '872460.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.991', 'adc_vl_f': '14.34', 'adc_ic_f': '1.285', 'adc_il_f': '0.314', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.435', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872464410 ms)
2025-02-17 22:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:15:25.400187+00:00', 'runtime': '872400.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.909', 'adc_vl_f': '14.34', 'adc_ic_f': '1.326', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.054', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872464410 ms)
2025-02-17 22:16:28.584 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+872466403 ms)
2025-02-17 22:16:28.584 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+872466403 ms)
2025-02-17 22:16:28.585 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38844)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+872466404 ms)
2025-02-17 22:17:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872524409 ms)
2025-02-17 22:17:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872524409 ms)
2025-02-17 22:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:17:25.400259+00:00', 'runtime': '872520.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.988', 'adc_vl_f': '14.34', 'adc_ic_f': '1.331', 'adc_il_f': '0.319', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.024', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872524411 ms)
2025-02-17 22:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:16:25.400199+00:00', 'runtime': '872460.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.991', 'adc_vl_f': '14.34', 'adc_ic_f': '1.285', 'adc_il_f': '0.314', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.435', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872524411 ms)
2025-02-17 22:17:35.885 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872533705 ms)
2025-02-17 22:17:35.886 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872533705 ms)
2025-02-17 22:17:35.887 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:17:25.400346+00:00'} (T+872533707 ms)
2025-02-17 22:17:40.744 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+872538564 ms)
2025-02-17 22:17:40.745 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+872538564 ms)
2025-02-17 22:17:40.745 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38922)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+872538564 ms)
2025-02-17 22:18:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872584410 ms)
2025-02-17 22:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872584411 ms)
2025-02-17 22:18:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:18:25.400199+00:00', 'runtime': '872580.001', 'adc_vb_f': '14.359', 'adc_va_f': '21.701', 'adc_vl_f': '14.343', 'adc_ic_f': '1.522', 'adc_il_f': '0.336', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.625', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872584413 ms)
2025-02-17 22:18:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:17:25.400259+00:00', 'runtime': '872520.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.988', 'adc_vl_f': '14.34', 'adc_ic_f': '1.331', 'adc_il_f': '0.319', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.024', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872584413 ms)
2025-02-17 22:18:35.919 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872593738 ms)
2025-02-17 22:18:35.920 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872593739 ms)
2025-02-17 22:18:35.924 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:18:25.400434+00:00'} (T+872593743 ms)
2025-02-17 22:18:40.895 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+872598714 ms)
2025-02-17 22:18:40.895 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+872598715 ms)
2025-02-17 22:18:40.896 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38988)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+872598715 ms)
2025-02-17 22:19:25.640 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872643459 ms)
2025-02-17 22:19:25.640 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872643460 ms)
2025-02-17 22:19:25.644 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:19:25.400177+00:00', 'runtime': '872640.001', 'adc_vb_f': '14.355', 'adc_va_f': '21.793', 'adc_vl_f': '14.34', 'adc_ic_f': '1.449', 'adc_il_f': '0.355', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.141', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872643464 ms)
2025-02-17 22:19:25.645 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:18:25.400199+00:00', 'runtime': '872580.001', 'adc_vb_f': '14.359', 'adc_va_f': '21.701', 'adc_vl_f': '14.343', 'adc_ic_f': '1.522', 'adc_il_f': '0.336', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.625', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872643464 ms)
2025-02-17 22:19:35.883 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872653703 ms)
2025-02-17 22:19:35.884 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872653703 ms)
2025-02-17 22:19:35.889 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:19:25.400397+00:00'} (T+872653708 ms)
2025-02-17 22:20:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872704412 ms)
2025-02-17 22:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872704412 ms)
2025-02-17 22:20:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:20:25.400113+00:00', 'runtime': '872700.001', 'adc_vb_f': '14.352', 'adc_va_f': '21.933', 'adc_vl_f': '14.34', 'adc_ic_f': '1.338', 'adc_il_f': '0.365', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.16', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872704416 ms)
2025-02-17 22:20:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:19:25.400177+00:00', 'runtime': '872640.001', 'adc_vb_f': '14.355', 'adc_va_f': '21.793', 'adc_vl_f': '14.34', 'adc_ic_f': '1.449', 'adc_il_f': '0.355', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.3', 'ahc_t': '2659.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.141', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.3', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872704417 ms)
2025-02-17 22:20:35.750 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872713569 ms)
2025-02-17 22:20:35.751 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872713570 ms)
2025-02-17 22:20:35.755 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:20:25.400437+00:00'} (T+872713574 ms)
2025-02-17 22:21:25.644 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872763463 ms)
2025-02-17 22:21:25.645 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872763464 ms)
2025-02-17 22:21:25.646 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:21:25.400187+00:00', 'runtime': '872760.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.93', 'adc_vl_f': '14.34', 'adc_ic_f': '1.321', 'adc_il_f': '0.353', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.919', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872763466 ms)
2025-02-17 22:21:25.647 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:20:25.400113+00:00', 'runtime': '872700.001', 'adc_vb_f': '14.352', 'adc_va_f': '21.933', 'adc_vl_f': '14.34', 'adc_ic_f': '1.338', 'adc_il_f': '0.365', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.16', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872763466 ms)
2025-02-17 22:21:56.237 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+872794056 ms)
2025-02-17 22:21:56.237 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+872794056 ms)
2025-02-17 22:21:56.237 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39198)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+872794057 ms)
2025-02-17 22:22:17.332 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+872815152 ms)
2025-02-17 22:22:17.333 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+872815152 ms)
2025-02-17 22:22:17.333 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39220)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+872815152 ms)
2025-02-17 22:22:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872824402 ms)
2025-02-17 22:22:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872824403 ms)
2025-02-17 22:22:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:22:25.400135+00:00', 'runtime': '872820.001', 'adc_vb_f': '14.352', 'adc_va_f': '21.906', 'adc_vl_f': '14.34', 'adc_ic_f': '1.321', 'adc_il_f': '0.348', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.556', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872824405 ms)
2025-02-17 22:22:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:21:25.400187+00:00', 'runtime': '872760.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.93', 'adc_vl_f': '14.34', 'adc_ic_f': '1.321', 'adc_il_f': '0.353', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.919', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872824405 ms)
2025-02-17 22:23:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872884404 ms)
2025-02-17 22:23:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872884404 ms)
2025-02-17 22:23:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:23:25.400189+00:00', 'runtime': '872880.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.884', 'adc_vl_f': '14.34', 'adc_ic_f': '1.314', 'adc_il_f': '0.326', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.873', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872884406 ms)
2025-02-17 22:23:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:22:25.400135+00:00', 'runtime': '872820.001', 'adc_vb_f': '14.352', 'adc_va_f': '21.906', 'adc_vl_f': '14.34', 'adc_ic_f': '1.321', 'adc_il_f': '0.348', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.556', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872884406 ms)
2025-02-17 22:23:35.778 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872893597 ms)
2025-02-17 22:23:35.778 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+872893598 ms)
2025-02-17 22:23:35.783 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:23:25.400338+00:00'} (T+872893603 ms)
2025-02-17 22:24:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+872944409 ms)
2025-02-17 22:24:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+872944410 ms)
2025-02-17 22:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:24:25.400206+00:00', 'runtime': '872940.001', 'adc_vb_f': '14.355', 'adc_va_f': '21.872', 'adc_vl_f': '14.34', 'adc_ic_f': '1.329', 'adc_il_f': '0.321', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.858', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872944414 ms)
2025-02-17 22:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:23:25.400189+00:00', 'runtime': '872880.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.884', 'adc_vl_f': '14.34', 'adc_ic_f': '1.314', 'adc_il_f': '0.326', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.4', 'ahc_t': '2659.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.873', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.4', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+872944414 ms)
2025-02-17 22:25:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873004411 ms)
2025-02-17 22:25:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873004411 ms)
2025-02-17 22:25:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:25:25.400176+00:00', 'runtime': '873000.001', 'adc_vb_f': '14.355', 'adc_va_f': '21.878', 'adc_vl_f': '14.34', 'adc_ic_f': '1.302', 'adc_il_f': '0.295', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.647', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873004415 ms)
2025-02-17 22:25:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:24:25.400206+00:00', 'runtime': '872940.001', 'adc_vb_f': '14.355', 'adc_va_f': '21.872', 'adc_vl_f': '14.34', 'adc_ic_f': '1.329', 'adc_il_f': '0.321', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.858', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873004415 ms)
2025-02-17 22:26:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873064413 ms)
2025-02-17 22:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873064413 ms)
2025-02-17 22:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:26:25.400176+00:00', 'runtime': '873060.001', 'adc_vb_f': '14.349', 'adc_va_f': '21.799', 'adc_vl_f': '14.34', 'adc_ic_f': '1.326', 'adc_il_f': '0.348', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.024', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873064415 ms)
2025-02-17 22:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:25:25.400176+00:00', 'runtime': '873000.001', 'adc_vb_f': '14.355', 'adc_va_f': '21.878', 'adc_vl_f': '14.34', 'adc_ic_f': '1.302', 'adc_il_f': '0.295', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.647', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873064416 ms)
2025-02-17 22:27:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873124415 ms)
2025-02-17 22:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873124416 ms)
2025-02-17 22:27:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:27:25.400165+00:00', 'runtime': '873120.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.741', 'adc_vl_f': '14.34', 'adc_ic_f': '1.305', 'adc_il_f': '0.297', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.737', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873124418 ms)
2025-02-17 22:27:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:26:25.400176+00:00', 'runtime': '873060.001', 'adc_vb_f': '14.349', 'adc_va_f': '21.799', 'adc_vl_f': '14.34', 'adc_ic_f': '1.326', 'adc_il_f': '0.348', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.6', 'ahl_t': '1592.6', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.024', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873124418 ms)
2025-02-17 22:28:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873184418 ms)
2025-02-17 22:28:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873184418 ms)
2025-02-17 22:28:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:28:25.400191+00:00', 'runtime': '873180.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.89', 'adc_vl_f': '14.34', 'adc_ic_f': '1.312', 'adc_il_f': '0.358', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.813', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873184420 ms)
2025-02-17 22:28:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:27:25.400165+00:00', 'runtime': '873120.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.741', 'adc_vl_f': '14.34', 'adc_ic_f': '1.305', 'adc_il_f': '0.297', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.737', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873184421 ms)
2025-02-17 22:28:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873193435 ms)
2025-02-17 22:28:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873193436 ms)
2025-02-17 22:28:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:28:25.400347+00:00'} (T+873193437 ms)
2025-02-17 22:28:41.889 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873199708 ms)
2025-02-17 22:28:41.889 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873199708 ms)
2025-02-17 22:28:41.890 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39636)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873199709 ms)
2025-02-17 22:28:55.984 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873213803 ms)
2025-02-17 22:28:55.984 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873213803 ms)
2025-02-17 22:28:55.984 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39654)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873213804 ms)
2025-02-17 22:29:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873244403 ms)
2025-02-17 22:29:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873244404 ms)
2025-02-17 22:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:29:25.400161+00:00', 'runtime': '873240.001', 'adc_vb_f': '14.349', 'adc_va_f': '21.765', 'adc_vl_f': '14.34', 'adc_ic_f': '1.283', 'adc_il_f': '0.35', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.36', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873244408 ms)
2025-02-17 22:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:28:25.400191+00:00', 'runtime': '873180.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.89', 'adc_vl_f': '14.34', 'adc_ic_f': '1.312', 'adc_il_f': '0.358', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.5', 'ahc_t': '2659.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.813', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.5', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873244409 ms)
2025-02-17 22:30:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873304408 ms)
2025-02-17 22:30:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873304409 ms)
2025-02-17 22:30:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:30:25.400209+00:00', 'runtime': '873300.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.768', 'adc_vl_f': '14.34', 'adc_ic_f': '1.276', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.254', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873304413 ms)
2025-02-17 22:30:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:29:25.400161+00:00', 'runtime': '873240.001', 'adc_vb_f': '14.349', 'adc_va_f': '21.765', 'adc_vl_f': '14.34', 'adc_ic_f': '1.283', 'adc_il_f': '0.35', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.36', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873304414 ms)
2025-02-17 22:31:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873364407 ms)
2025-02-17 22:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873364408 ms)
2025-02-17 22:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:31:25.400611+00:00', 'runtime': '873360.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.878', 'adc_vl_f': '14.34', 'adc_ic_f': '1.3', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.601', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873364412 ms)
2025-02-17 22:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:30:25.400209+00:00', 'runtime': '873300.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.768', 'adc_vl_f': '14.34', 'adc_ic_f': '1.276', 'adc_il_f': '0.362', 't_hs': '22', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.254', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873364412 ms)
2025-02-17 22:32:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873424410 ms)
2025-02-17 22:32:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873424410 ms)
2025-02-17 22:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:32:25.402800+00:00', 'runtime': '873420.003', 'adc_vb_f': '14.346', 'adc_va_f': '21.878', 'adc_vl_f': '14.34', 'adc_ic_f': '1.302', 'adc_il_f': '0.35', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.632', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873424414 ms)
2025-02-17 22:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:31:25.400611+00:00', 'runtime': '873360.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.878', 'adc_vl_f': '14.34', 'adc_ic_f': '1.3', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.346', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.601', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873424415 ms)
2025-02-17 22:32:35.728 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873433548 ms)
2025-02-17 22:32:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873433548 ms)
2025-02-17 22:32:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:32:25.400136+00:00'} (T+873433552 ms)
2025-02-17 22:32:57.375 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873455194 ms)
2025-02-17 22:32:57.375 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873455195 ms)
2025-02-17 22:32:57.376 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39908)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873455195 ms)
2025-02-17 22:33:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873484414 ms)
2025-02-17 22:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873484415 ms)
2025-02-17 22:33:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:33:25.400052+00:00', 'runtime': '873480.001', 'adc_vb_f': '14.352', 'adc_va_f': '21.786', 'adc_vl_f': '14.34', 'adc_ic_f': '1.302', 'adc_il_f': '0.319', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.647', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873484418 ms)
2025-02-17 22:33:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:32:25.402800+00:00', 'runtime': '873420.003', 'adc_vb_f': '14.346', 'adc_va_f': '21.878', 'adc_vl_f': '14.34', 'adc_ic_f': '1.302', 'adc_il_f': '0.35', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.6', 'ahc_t': '2659.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.632', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.6', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873484418 ms)
2025-02-17 22:34:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873544416 ms)
2025-02-17 22:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873544416 ms)
2025-02-17 22:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:34:25.400070+00:00', 'runtime': '873540.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.78', 'adc_vl_f': '14.34', 'adc_ic_f': '1.271', 'adc_il_f': '0.314', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.224', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873544417 ms)
2025-02-17 22:34:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:33:25.400052+00:00', 'runtime': '873480.001', 'adc_vb_f': '14.352', 'adc_va_f': '21.786', 'adc_vl_f': '14.34', 'adc_ic_f': '1.302', 'adc_il_f': '0.319', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.647', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873544420 ms)
2025-02-17 22:35:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873604419 ms)
2025-02-17 22:35:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873604419 ms)
2025-02-17 22:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:35:25.400698+00:00', 'runtime': '873600.001', 'adc_vb_f': '14.349', 'adc_va_f': '21.771', 'adc_vl_f': '14.34', 'adc_ic_f': '1.297', 'adc_il_f': '0.304', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.601', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873604420 ms)
2025-02-17 22:35:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:34:25.400070+00:00', 'runtime': '873540.001', 'adc_vb_f': '14.346', 'adc_va_f': '21.78', 'adc_vl_f': '14.34', 'adc_ic_f': '1.271', 'adc_il_f': '0.314', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.224', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873604423 ms)
2025-02-17 22:35:50.667 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873628487 ms)
2025-02-17 22:35:50.668 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873628487 ms)
2025-02-17 22:35:50.668 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40100)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873628488 ms)
2025-02-17 22:36:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873664418 ms)
2025-02-17 22:36:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873664419 ms)
2025-02-17 22:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:36:25.402774+00:00', 'runtime': '873660.003', 'adc_vb_f': '14.389', 'adc_va_f': '21.701', 'adc_vl_f': '14.377', 'adc_ic_f': '1.3', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.383', 'vb_ref': '13.912', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.967', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873664421 ms)
2025-02-17 22:36:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:35:25.400698+00:00', 'runtime': '873600.001', 'adc_vb_f': '14.349', 'adc_va_f': '21.771', 'adc_vl_f': '14.34', 'adc_ic_f': '1.297', 'adc_il_f': '0.304', 't_hs': '21', 't_batt': '23', 't_amb': '23', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '13.883', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.601', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873664421 ms)
2025-02-17 22:36:33.554 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: connect to host hamma.dev port 22: Network is unreachable
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
 (T+873671373 ms)
2025-02-17 22:37:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873724407 ms)
2025-02-17 22:37:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873724408 ms)
2025-02-17 22:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:37:25.400153+00:00', 'runtime': '873720.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.664', 'adc_vl_f': '14.377', 'adc_ic_f': '1.266', 'adc_il_f': '0.304', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.164', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873724409 ms)
2025-02-17 22:37:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:36:25.402774+00:00', 'runtime': '873660.003', 'adc_vb_f': '14.389', 'adc_va_f': '21.701', 'adc_vl_f': '14.377', 'adc_ic_f': '1.3', 'adc_il_f': '0.326', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.383', 'vb_ref': '13.912', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.967', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873724410 ms)
2025-02-17 22:37:46.892 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873744712 ms)
2025-02-17 22:37:46.893 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873744712 ms)
2025-02-17 22:37:46.893 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40220)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873744712 ms)
2025-02-17 22:37:54.972 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873752791 ms)
2025-02-17 22:37:54.972 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873752791 ms)
2025-02-17 22:37:54.973 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40232)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873752792 ms)
2025-02-17 22:38:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873784413 ms)
2025-02-17 22:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873784414 ms)
2025-02-17 22:38:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:38:25.400184+00:00', 'runtime': '873780.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.674', 'adc_vl_f': '14.377', 'adc_ic_f': '1.259', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.103', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873784415 ms)
2025-02-17 22:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:37:25.400153+00:00', 'runtime': '873720.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.664', 'adc_vl_f': '14.377', 'adc_ic_f': '1.266', 'adc_il_f': '0.304', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2659.7', 'ahc_t': '2659.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.164', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.7', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873784416 ms)
2025-02-17 22:38:35.633 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873793452 ms)
2025-02-17 22:38:35.634 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873793453 ms)
2025-02-17 22:38:35.640 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:38:25.405071+00:00'} (T+873793460 ms)
2025-02-17 22:38:57.113 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873814933 ms)
2025-02-17 22:38:57.114 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873814933 ms)
2025-02-17 22:38:57.114 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40300)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873814933 ms)
2025-02-17 22:39:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873844414 ms)
2025-02-17 22:39:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873844415 ms)
2025-02-17 22:39:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:39:25.402796+00:00', 'runtime': '873840.003', 'adc_vb_f': '14.386', 'adc_va_f': '21.664', 'adc_vl_f': '14.377', 'adc_ic_f': '1.244', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.816', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873844418 ms)
2025-02-17 22:39:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:38:25.400184+00:00', 'runtime': '873780.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.674', 'adc_vl_f': '14.377', 'adc_ic_f': '1.259', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.103', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873844419 ms)
2025-02-17 22:40:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873904404 ms)
2025-02-17 22:40:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873904405 ms)
2025-02-17 22:40:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:40:25.400191+00:00', 'runtime': '873900.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.597', 'adc_vl_f': '14.377', 'adc_ic_f': '1.329', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.784', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873904409 ms)
2025-02-17 22:40:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:39:25.402796+00:00', 'runtime': '873840.003', 'adc_vb_f': '14.386', 'adc_va_f': '21.664', 'adc_vl_f': '14.377', 'adc_ic_f': '1.244', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.816', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873904409 ms)
2025-02-17 22:41:24.359 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+873962178 ms)
2025-02-17 22:41:24.359 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+873962179 ms)
2025-02-17 22:41:24.360 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40454)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+873962179 ms)
2025-02-17 22:41:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+873964410 ms)
2025-02-17 22:41:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+873964411 ms)
2025-02-17 22:41:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:41:25.400194+00:00', 'runtime': '873960.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.576', 'adc_vl_f': '14.377', 'adc_ic_f': '1.259', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.997', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873964414 ms)
2025-02-17 22:41:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:40:25.400191+00:00', 'runtime': '873900.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.597', 'adc_vl_f': '14.377', 'adc_ic_f': '1.329', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.386', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.784', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+873964414 ms)
2025-02-17 22:41:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873973434 ms)
2025-02-17 22:41:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+873973435 ms)
2025-02-17 22:41:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:41:25.405064+00:00'} (T+873973437 ms)
2025-02-17 22:42:25.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874023402 ms)
2025-02-17 22:42:25.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874023403 ms)
2025-02-17 22:42:25.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:42:25.401180+00:00', 'runtime': '874020.002', 'adc_vb_f': '14.386', 'adc_va_f': '21.448', 'adc_vl_f': '14.377', 'adc_ic_f': '1.239', 'adc_il_f': '0.329', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.68', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874023407 ms)
2025-02-17 22:42:25.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:41:25.400194+00:00', 'runtime': '873960.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.576', 'adc_vl_f': '14.377', 'adc_ic_f': '1.259', 'adc_il_f': '0.333', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2659.8', 'ahc_t': '2659.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.997', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.8', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874023407 ms)
2025-02-17 22:42:35.939 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874033758 ms)
2025-02-17 22:42:35.940 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874033759 ms)
2025-02-17 22:42:35.944 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:42:25.400527+00:00'} (T+874033763 ms)
2025-02-17 22:43:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874084416 ms)
2025-02-17 22:43:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874084416 ms)
2025-02-17 22:43:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:43:25.402915+00:00', 'runtime': '874080.003', 'adc_vb_f': '14.34', 'adc_va_f': '19.919', 'adc_vl_f': '14.301', 'adc_ic_f': '1.305', 'adc_il_f': '0.321', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.377', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.662', 'sweep_vmp': '19.943', 'sweep_pmax': '18.994', 'sweep_voc': '22.266', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874084420 ms)
2025-02-17 22:43:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:42:25.401180+00:00', 'runtime': '874020.002', 'adc_vb_f': '14.386', 'adc_va_f': '21.448', 'adc_vl_f': '14.377', 'adc_ic_f': '1.239', 'adc_il_f': '0.329', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.389', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '17.68', 'sweep_vmp': '20.737', 'sweep_pmax': '98.79', 'sweep_voc': '23.181', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874084421 ms)
2025-02-17 22:43:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874093423 ms)
2025-02-17 22:43:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874093423 ms)
2025-02-17 22:43:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:43:25.400275+00:00'} (T+874093427 ms)
2025-02-17 22:43:36.592 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+874094411 ms)
2025-02-17 22:43:36.592 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+874094411 ms)
2025-02-17 22:43:36.592 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40598)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+874094412 ms)
2025-02-17 22:44:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874144419 ms)
2025-02-17 22:44:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874144419 ms)
2025-02-17 22:44:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:44:25.400513+00:00', 'runtime': '874140.001', 'adc_vb_f': '14.386', 'adc_va_f': '19.925', 'adc_vl_f': '14.34', 'adc_ic_f': '1.28', 'adc_il_f': '0.355', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.355', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.405', 'sweep_vmp': '19.943', 'sweep_pmax': '18.994', 'sweep_voc': '22.266', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874144421 ms)
2025-02-17 22:44:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:43:25.402915+00:00', 'runtime': '874080.003', 'adc_vb_f': '14.34', 'adc_va_f': '19.919', 'adc_vl_f': '14.301', 'adc_ic_f': '1.305', 'adc_il_f': '0.321', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.377', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.662', 'sweep_vmp': '19.943', 'sweep_pmax': '18.994', 'sweep_voc': '22.266', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874144422 ms)
2025-02-17 22:45:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874204403 ms)
2025-02-17 22:45:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874204404 ms)
2025-02-17 22:45:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:45:25.400066+00:00', 'runtime': '874200.001', 'adc_vb_f': '14.389', 'adc_va_f': '19.598', 'adc_vl_f': '14.343', 'adc_ic_f': '1.309', 'adc_il_f': '0.316', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.145', 'sweep_vmp': '19.943', 'sweep_pmax': '18.994', 'sweep_voc': '22.266', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874204406 ms)
2025-02-17 22:45:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:44:25.400513+00:00', 'runtime': '874140.001', 'adc_vb_f': '14.386', 'adc_va_f': '19.925', 'adc_vl_f': '14.34', 'adc_ic_f': '1.28', 'adc_il_f': '0.355', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.355', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.405', 'sweep_vmp': '19.943', 'sweep_pmax': '18.994', 'sweep_voc': '22.266', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874204406 ms)
2025-02-17 22:46:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874264405 ms)
2025-02-17 22:46:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874264405 ms)
2025-02-17 22:46:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:46:25.400651+00:00', 'runtime': '874260.001', 'adc_vb_f': '14.31', 'adc_va_f': '19.87', 'adc_vl_f': '14.264', 'adc_ic_f': '1.329', 'adc_il_f': '0.324', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.371', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.964', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874264408 ms)
2025-02-17 22:46:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:45:25.400066+00:00', 'runtime': '874200.001', 'adc_vb_f': '14.389', 'adc_va_f': '19.598', 'adc_vl_f': '14.343', 'adc_ic_f': '1.309', 'adc_il_f': '0.316', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.325', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.145', 'sweep_vmp': '19.943', 'sweep_pmax': '18.994', 'sweep_voc': '22.266', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874264408 ms)
2025-02-17 22:47:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874324422 ms)
2025-02-17 22:47:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874324423 ms)
2025-02-17 22:47:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:47:25.400183+00:00', 'runtime': '874320.001', 'adc_vb_f': '14.395', 'adc_va_f': '19.87', 'adc_vl_f': '14.34', 'adc_ic_f': '1.319', 'adc_il_f': '0.329', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.374', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.979', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874324425 ms)
2025-02-17 22:47:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:46:25.400651+00:00', 'runtime': '874260.001', 'adc_vb_f': '14.31', 'adc_va_f': '19.87', 'adc_vl_f': '14.264', 'adc_ic_f': '1.329', 'adc_il_f': '0.324', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.371', 'vb_ref': '13.912', 'ahc_r': '2659.9', 'ahc_t': '2659.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.7', 'ahl_t': '1592.7', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.964', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '29.9', 'ahl_daily': '8.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874324425 ms)
2025-02-17 22:48:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874384411 ms)
2025-02-17 22:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874384412 ms)
2025-02-17 22:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:48:25.400858+00:00', 'runtime': '874380.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.127', 'adc_vl_f': '14.377', 'adc_ic_f': '1.28', 'adc_il_f': '0.345', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.337', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.315', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874384414 ms)
2025-02-17 22:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:47:25.400183+00:00', 'runtime': '874320.001', 'adc_vb_f': '14.395', 'adc_va_f': '19.87', 'adc_vl_f': '14.34', 'adc_ic_f': '1.319', 'adc_il_f': '0.329', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.374', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.979', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874384414 ms)
2025-02-17 22:49:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874444415 ms)
2025-02-17 22:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874444416 ms)
2025-02-17 22:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:49:25.400626+00:00', 'runtime': '874440.001', 'adc_vb_f': '14.386', 'adc_va_f': '19.733', 'adc_vl_f': '14.34', 'adc_ic_f': '1.321', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.383', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.1', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874444417 ms)
2025-02-17 22:49:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:48:25.400858+00:00', 'runtime': '874380.001', 'adc_vb_f': '14.386', 'adc_va_f': '21.127', 'adc_vl_f': '14.377', 'adc_ic_f': '1.28', 'adc_il_f': '0.345', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.337', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '18.315', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874444418 ms)
2025-02-17 22:50:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874504416 ms)
2025-02-17 22:50:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874504416 ms)
2025-02-17 22:50:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:50:25.402786+00:00', 'runtime': '874500.003', 'adc_vb_f': '14.368', 'adc_va_f': '20.02', 'adc_vl_f': '14.325', 'adc_ic_f': '1.263', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.383', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.088', 'sweep_vmp': '20.044', 'sweep_pmax': '19.522', 'sweep_voc': '22.412', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874504418 ms)
2025-02-17 22:50:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:49:25.400626+00:00', 'runtime': '874440.001', 'adc_vb_f': '14.386', 'adc_va_f': '19.733', 'adc_vl_f': '14.34', 'adc_ic_f': '1.321', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.383', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '19.1', 'sweep_vmp': '19.919', 'sweep_pmax': '19.251', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874504419 ms)
2025-02-17 22:51:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874564418 ms)
2025-02-17 22:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874564419 ms)
2025-02-17 22:51:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:51:25.400185+00:00', 'runtime': '874560.001', 'adc_vb_f': '14.194', 'adc_va_f': '20.02', 'adc_vl_f': '14.175', 'adc_ic_f': '1.288', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.163', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.269', 'sweep_vmp': '20.044', 'sweep_pmax': '19.522', 'sweep_voc': '22.412', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874564420 ms)
2025-02-17 22:51:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:50:25.402786+00:00', 'runtime': '874500.003', 'adc_vb_f': '14.368', 'adc_va_f': '20.02', 'adc_vl_f': '14.325', 'adc_ic_f': '1.263', 'adc_il_f': '0.36', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.383', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.088', 'sweep_vmp': '20.044', 'sweep_pmax': '19.522', 'sweep_voc': '22.412', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874564421 ms)
2025-02-17 22:52:25.635 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874623454 ms)
2025-02-17 22:52:25.635 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874623455 ms)
2025-02-17 22:52:25.637 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:52:25.400194+00:00', 'runtime': '874620.001', 'adc_vb_f': '14.148', 'adc_va_f': '20.02', 'adc_vl_f': '14.111', 'adc_ic_f': '1.244', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.16', 'vb_ref': '13.912', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.62', 'sweep_vmp': '20.044', 'sweep_pmax': '19.522', 'sweep_voc': '22.412', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874623456 ms)
2025-02-17 22:52:25.638 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:51:25.400185+00:00', 'runtime': '874560.001', 'adc_vb_f': '14.194', 'adc_va_f': '20.02', 'adc_vl_f': '14.175', 'adc_ic_f': '1.288', 'adc_il_f': '0.362', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.163', 'vb_ref': '13.912', 'ahc_r': '2660.0', 'ahc_t': '2660.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.269', 'sweep_vmp': '20.044', 'sweep_pmax': '19.522', 'sweep_voc': '22.412', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.0', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874623457 ms)
2025-02-17 22:52:35.646 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874633465 ms)
2025-02-17 22:52:35.646 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874633466 ms)
2025-02-17 22:52:35.648 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:52:25.400336+00:00'} (T+874633467 ms)
2025-02-17 22:53:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874684408 ms)
2025-02-17 22:53:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874684409 ms)
2025-02-17 22:53:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:53:25.400158+00:00', 'runtime': '874680.001', 'adc_vb_f': '13.928', 'adc_va_f': '20.016', 'adc_vl_f': '13.919', 'adc_ic_f': '1.203', 'adc_il_f': '0.355', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.983', 'vb_ref': '13.912', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.714', 'sweep_vmp': '20.029', 'sweep_pmax': '17.831', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874684410 ms)
2025-02-17 22:53:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:52:25.400194+00:00', 'runtime': '874620.001', 'adc_vb_f': '14.148', 'adc_va_f': '20.02', 'adc_vl_f': '14.111', 'adc_ic_f': '1.244', 'adc_il_f': '0.341', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.16', 'vb_ref': '13.912', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.62', 'sweep_vmp': '20.044', 'sweep_pmax': '19.522', 'sweep_voc': '22.412', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874684410 ms)
2025-02-17 22:53:35.925 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874693744 ms)
2025-02-17 22:53:35.925 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874693745 ms)
2025-02-17 22:53:35.930 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:53:25.400175+00:00'} (T+874693749 ms)
2025-02-17 22:54:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874744410 ms)
2025-02-17 22:54:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874744411 ms)
2025-02-17 22:54:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:54:25.400197+00:00', 'runtime': '874740.001', 'adc_vb_f': '13.66', 'adc_va_f': '20.016', 'adc_vl_f': '13.614', 'adc_ic_f': '1.176', 'adc_il_f': '0.358', 't_hs': '21', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.727', 'vb_ref': '13.942', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.065', 'sweep_vmp': '20.029', 'sweep_pmax': '17.831', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874744412 ms)
2025-02-17 22:54:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:53:25.400158+00:00', 'runtime': '874680.001', 'adc_vb_f': '13.928', 'adc_va_f': '20.016', 'adc_vl_f': '13.919', 'adc_ic_f': '1.203', 'adc_il_f': '0.355', 't_hs': '21', 't_batt': '22', 't_amb': '22', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.983', 'vb_ref': '13.912', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20129', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.714', 'sweep_vmp': '20.029', 'sweep_pmax': '17.831', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874744414 ms)
2025-02-17 22:54:35.729 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874753548 ms)
2025-02-17 22:54:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874753549 ms)
2025-02-17 22:54:35.731 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:54:25.400389+00:00'} (T+874753551 ms)
2025-02-17 22:55:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874804410 ms)
2025-02-17 22:55:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874804411 ms)
2025-02-17 22:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:55:25.400211+00:00', 'runtime': '874800.001', 'adc_vb_f': '13.586', 'adc_va_f': '20.02', 'adc_vl_f': '13.577', 'adc_ic_f': '1.155', 'adc_il_f': '0.345', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.605', 'vb_ref': '13.942', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.687', 'sweep_vmp': '20.029', 'sweep_pmax': '17.831', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874804412 ms)
2025-02-17 22:55:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:54:25.400197+00:00', 'runtime': '874740.001', 'adc_vb_f': '13.66', 'adc_va_f': '20.016', 'adc_vl_f': '13.614', 'adc_ic_f': '1.176', 'adc_il_f': '0.358', 't_hs': '21', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.727', 'vb_ref': '13.942', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.065', 'sweep_vmp': '20.029', 'sweep_pmax': '17.831', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874804412 ms)
2025-02-17 22:56:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874864414 ms)
2025-02-17 22:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874864414 ms)
2025-02-17 22:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:56:25.400187+00:00', 'runtime': '874860.001', 'adc_vb_f': '13.434', 'adc_va_f': '19.873', 'adc_vl_f': '13.416', 'adc_ic_f': '1.036', 'adc_il_f': '0.324', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.467', 'vb_ref': '13.942', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.876', 'sweep_vmp': '19.91', 'sweep_pmax': '15.748', 'sweep_voc': '22.302', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874864415 ms)
2025-02-17 22:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:55:25.400211+00:00', 'runtime': '874800.001', 'adc_vb_f': '13.586', 'adc_va_f': '20.02', 'adc_vl_f': '13.577', 'adc_ic_f': '1.155', 'adc_il_f': '0.345', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.605', 'vb_ref': '13.942', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.687', 'sweep_vmp': '20.029', 'sweep_pmax': '17.831', 'sweep_voc': '22.321', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874864416 ms)
2025-02-17 22:56:46.601 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+874884420 ms)
2025-02-17 22:56:46.601 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+874884421 ms)
2025-02-17 22:56:46.602 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41448)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+874884421 ms)
2025-02-17 22:57:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874924414 ms)
2025-02-17 22:57:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874924414 ms)
2025-02-17 22:57:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:57:25.400166+00:00', 'runtime': '874920.001', 'adc_vb_f': '13.412', 'adc_va_f': '19.876', 'adc_vl_f': '13.385', 'adc_ic_f': '1.041', 'adc_il_f': '0.343', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.412', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.951', 'sweep_vmp': '19.91', 'sweep_pmax': '15.748', 'sweep_voc': '22.302', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874924416 ms)
2025-02-17 22:57:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:56:25.400187+00:00', 'runtime': '874860.001', 'adc_vb_f': '13.434', 'adc_va_f': '19.873', 'adc_vl_f': '13.416', 'adc_ic_f': '1.036', 'adc_il_f': '0.324', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.467', 'vb_ref': '13.942', 'ahc_r': '2660.1', 'ahc_t': '2660.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.876', 'sweep_vmp': '19.91', 'sweep_pmax': '15.748', 'sweep_voc': '22.302', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.1', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874924417 ms)
2025-02-17 22:57:35.627 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874933447 ms)
2025-02-17 22:57:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+874933447 ms)
2025-02-17 22:57:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 22:57:25.400683+00:00'} (T+874933449 ms)
2025-02-17 22:58:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+874984418 ms)
2025-02-17 22:58:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+874984418 ms)
2025-02-17 22:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:58:25.400191+00:00', 'runtime': '874980.001', 'adc_vb_f': '13.397', 'adc_va_f': '19.87', 'adc_vl_f': '13.385', 'adc_ic_f': '1.034', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.4', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.845', 'sweep_vmp': '19.91', 'sweep_pmax': '15.748', 'sweep_voc': '22.302', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874984419 ms)
2025-02-17 22:58:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:57:25.400166+00:00', 'runtime': '874920.001', 'adc_vb_f': '13.412', 'adc_va_f': '19.876', 'adc_vl_f': '13.385', 'adc_ic_f': '1.041', 'adc_il_f': '0.343', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.412', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.951', 'sweep_vmp': '19.91', 'sweep_pmax': '15.748', 'sweep_voc': '22.302', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+874984420 ms)
2025-02-17 22:59:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875044404 ms)
2025-02-17 22:59:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875044404 ms)
2025-02-17 22:59:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 22:59:25.400159+00:00', 'runtime': '875040.001', 'adc_vb_f': '13.385', 'adc_va_f': '19.925', 'adc_vl_f': '13.373', 'adc_ic_f': '1.007', 'adc_il_f': '0.345', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.388', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.513', 'sweep_vmp': '19.928', 'sweep_pmax': '14.132', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875044405 ms)
2025-02-17 22:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:58:25.400191+00:00', 'runtime': '874980.001', 'adc_vb_f': '13.397', 'adc_va_f': '19.87', 'adc_vl_f': '13.385', 'adc_ic_f': '1.034', 'adc_il_f': '0.341', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.4', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.845', 'sweep_vmp': '19.91', 'sweep_pmax': '15.748', 'sweep_voc': '22.302', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875044406 ms)
2025-02-17 23:00:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875104406 ms)
2025-02-17 23:00:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875104406 ms)
2025-02-17 23:00:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:00:25.400188+00:00', 'runtime': '875100.001', 'adc_vb_f': '13.358', 'adc_va_f': '19.925', 'adc_vl_f': '13.312', 'adc_ic_f': '0.974', 'adc_il_f': '0.384', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.361', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.939', 'sweep_vmp': '19.928', 'sweep_pmax': '14.132', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875104407 ms)
2025-02-17 23:00:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 22:59:25.400159+00:00', 'runtime': '875040.001', 'adc_vb_f': '13.385', 'adc_va_f': '19.925', 'adc_vl_f': '13.373', 'adc_ic_f': '1.007', 'adc_il_f': '0.345', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.388', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.513', 'sweep_vmp': '19.928', 'sweep_pmax': '14.132', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875104408 ms)
2025-02-17 23:01:22.994 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+875160813 ms)
2025-02-17 23:01:22.994 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+875160813 ms)
2025-02-17 23:01:22.994 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41742)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+875160813 ms)
2025-02-17 23:01:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875164408 ms)
2025-02-17 23:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875164408 ms)
2025-02-17 23:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:01:25.400190+00:00', 'runtime': '875160.001', 'adc_vb_f': '13.321', 'adc_va_f': '19.925', 'adc_vl_f': '13.309', 'adc_ic_f': '0.906', 'adc_il_f': '0.389', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.336', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.064', 'sweep_vmp': '19.928', 'sweep_pmax': '14.132', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875164410 ms)
2025-02-17 23:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:00:25.400188+00:00', 'runtime': '875100.001', 'adc_vb_f': '13.358', 'adc_va_f': '19.925', 'adc_vl_f': '13.312', 'adc_ic_f': '0.974', 'adc_il_f': '0.384', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.361', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.939', 'sweep_vmp': '19.928', 'sweep_pmax': '14.132', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875164410 ms)
2025-02-17 23:02:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875224412 ms)
2025-02-17 23:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875224412 ms)
2025-02-17 23:02:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:02:25.400178+00:00', 'runtime': '875220.001', 'adc_vb_f': '13.281', 'adc_va_f': '19.733', 'adc_vl_f': '13.266', 'adc_ic_f': '0.797', 'adc_il_f': '0.387', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.287', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.584', 'sweep_vmp': '19.754', 'sweep_pmax': '12.018', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875224414 ms)
2025-02-17 23:02:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:01:25.400190+00:00', 'runtime': '875160.001', 'adc_vb_f': '13.321', 'adc_va_f': '19.925', 'adc_vl_f': '13.309', 'adc_ic_f': '0.906', 'adc_il_f': '0.389', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.336', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.064', 'sweep_vmp': '19.928', 'sweep_pmax': '14.132', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875224414 ms)
2025-02-17 23:03:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875284414 ms)
2025-02-17 23:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875284414 ms)
2025-02-17 23:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:03:25.400197+00:00', 'runtime': '875280.001', 'adc_vb_f': '13.245', 'adc_va_f': '19.733', 'adc_vl_f': '13.229', 'adc_ic_f': '0.73', 'adc_il_f': '0.377', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.251', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.603', 'sweep_vmp': '19.754', 'sweep_pmax': '12.018', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875284415 ms)
2025-02-17 23:03:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:02:25.400178+00:00', 'runtime': '875220.001', 'adc_vb_f': '13.281', 'adc_va_f': '19.733', 'adc_vl_f': '13.266', 'adc_ic_f': '0.797', 'adc_il_f': '0.387', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.287', 'vb_ref': '13.942', 'ahc_r': '2660.2', 'ahc_t': '2660.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.584', 'sweep_vmp': '19.754', 'sweep_pmax': '12.018', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.2', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875284416 ms)
2025-02-17 23:04:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875344415 ms)
2025-02-17 23:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875344416 ms)
2025-02-17 23:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:04:25.400191+00:00', 'runtime': '875340.001', 'adc_vb_f': '13.205', 'adc_va_f': '19.73', 'adc_vl_f': '13.156', 'adc_ic_f': '0.681', 'adc_il_f': '0.379', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.211', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.984', 'sweep_vmp': '19.754', 'sweep_pmax': '12.018', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875344417 ms)
2025-02-17 23:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:03:25.400197+00:00', 'runtime': '875280.001', 'adc_vb_f': '13.245', 'adc_va_f': '19.733', 'adc_vl_f': '13.229', 'adc_ic_f': '0.73', 'adc_il_f': '0.377', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.251', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.8', 'ahl_t': '1592.8', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.603', 'sweep_vmp': '19.754', 'sweep_pmax': '12.018', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875344417 ms)
2025-02-17 23:04:42.332 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+875360152 ms)
2025-02-17 23:04:42.333 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+875360152 ms)
2025-02-17 23:04:42.333 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41950)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+875360152 ms)
2025-02-17 23:05:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875404417 ms)
2025-02-17 23:05:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875404418 ms)
2025-02-17 23:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:05:25.400209+00:00', 'runtime': '875400.001', 'adc_vb_f': '13.196', 'adc_va_f': '19.681', 'adc_vl_f': '13.156', 'adc_ic_f': '0.703', 'adc_il_f': '0.365', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.199', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.24', 'sweep_vmp': '19.717', 'sweep_pmax': '9.21', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875404419 ms)
2025-02-17 23:05:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:04:25.400191+00:00', 'runtime': '875340.001', 'adc_vb_f': '13.205', 'adc_va_f': '19.73', 'adc_vl_f': '13.156', 'adc_ic_f': '0.681', 'adc_il_f': '0.379', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.211', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.984', 'sweep_vmp': '19.754', 'sweep_pmax': '12.018', 'sweep_voc': '22.192', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875404420 ms)
2025-02-17 23:06:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875464404 ms)
2025-02-17 23:06:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875464405 ms)
2025-02-17 23:06:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:06:25.400176+00:00', 'runtime': '875460.001', 'adc_vb_f': '13.168', 'adc_va_f': '19.681', 'adc_vl_f': '13.156', 'adc_ic_f': '0.669', 'adc_il_f': '0.35', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.171', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.818', 'sweep_vmp': '19.717', 'sweep_pmax': '9.21', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875464407 ms)
2025-02-17 23:06:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:05:25.400209+00:00', 'runtime': '875400.001', 'adc_vb_f': '13.196', 'adc_va_f': '19.681', 'adc_vl_f': '13.156', 'adc_ic_f': '0.703', 'adc_il_f': '0.365', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.199', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.24', 'sweep_vmp': '19.717', 'sweep_pmax': '9.21', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875464407 ms)
2025-02-17 23:06:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+875473423 ms)
2025-02-17 23:06:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+875473423 ms)
2025-02-17 23:06:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:06:25.400343+00:00'} (T+875473429 ms)
2025-02-17 23:07:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875524407 ms)
2025-02-17 23:07:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875524407 ms)
2025-02-17 23:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:07:25.400210+00:00', 'runtime': '875520.001', 'adc_vb_f': '13.138', 'adc_va_f': '19.684', 'adc_vl_f': '13.12', 'adc_ic_f': '0.616', 'adc_il_f': '0.343', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.153', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.078', 'sweep_vmp': '19.717', 'sweep_pmax': '9.21', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875524411 ms)
2025-02-17 23:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:06:25.400176+00:00', 'runtime': '875460.001', 'adc_vb_f': '13.168', 'adc_va_f': '19.681', 'adc_vl_f': '13.156', 'adc_ic_f': '0.669', 'adc_il_f': '0.35', 't_hs': '20', 't_batt': '21', 't_amb': '21', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.171', 'vb_ref': '13.942', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.818', 'sweep_vmp': '19.717', 'sweep_pmax': '9.21', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875524411 ms)
2025-02-17 23:08:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875584408 ms)
2025-02-17 23:08:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875584409 ms)
2025-02-17 23:08:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:08:25.400181+00:00', 'runtime': '875580.001', 'adc_vb_f': '13.129', 'adc_va_f': '19.58', 'adc_vl_f': '13.116', 'adc_ic_f': '0.582', 'adc_il_f': '0.333', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.132', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.579', 'sweep_vmp': '19.614', 'sweep_pmax': '8.168', 'sweep_voc': '22.003', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875584413 ms)
2025-02-17 23:08:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:07:25.400210+00:00', 'runtime': '875520.001', 'adc_vb_f': '13.138', 'adc_va_f': '19.684', 'adc_vl_f': '13.12', 'adc_ic_f': '0.616', 'adc_il_f': '0.343', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.153', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.078', 'sweep_vmp': '19.717', 'sweep_pmax': '9.21', 'sweep_voc': '22.095', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875584413 ms)
2025-02-17 23:08:27.722 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+875585541 ms)
2025-02-17 23:08:27.722 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+875585542 ms)
2025-02-17 23:08:27.723 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42198)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+875585542 ms)
2025-02-17 23:09:23.870 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+875641689 ms)
2025-02-17 23:09:23.870 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+875641690 ms)
2025-02-17 23:09:23.871 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42258)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+875641690 ms)
2025-02-17 23:09:26.608 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875644427 ms)
2025-02-17 23:09:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875644428 ms)
2025-02-17 23:09:26.612 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:09:25.400172+00:00', 'runtime': '875640.001', 'adc_vb_f': '13.098', 'adc_va_f': '19.583', 'adc_vl_f': '13.08', 'adc_ic_f': '0.568', 'adc_il_f': '0.367', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.113', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.428', 'sweep_vmp': '19.614', 'sweep_pmax': '8.168', 'sweep_voc': '22.003', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875644431 ms)
2025-02-17 23:09:26.613 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:08:25.400181+00:00', 'runtime': '875580.001', 'adc_vb_f': '13.129', 'adc_va_f': '19.58', 'adc_vl_f': '13.116', 'adc_ic_f': '0.582', 'adc_il_f': '0.333', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.132', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.579', 'sweep_vmp': '19.614', 'sweep_pmax': '8.168', 'sweep_voc': '22.003', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875644432 ms)
2025-02-17 23:10:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875704410 ms)
2025-02-17 23:10:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875704410 ms)
2025-02-17 23:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:10:25.400161+00:00', 'runtime': '875700.001', 'adc_vb_f': '13.092', 'adc_va_f': '19.583', 'adc_vl_f': '13.077', 'adc_ic_f': '0.534', 'adc_il_f': '0.365', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.095', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.976', 'sweep_vmp': '19.614', 'sweep_pmax': '8.168', 'sweep_voc': '22.003', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875704415 ms)
2025-02-17 23:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:09:25.400172+00:00', 'runtime': '875640.001', 'adc_vb_f': '13.098', 'adc_va_f': '19.583', 'adc_vl_f': '13.08', 'adc_ic_f': '0.568', 'adc_il_f': '0.367', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.113', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.428', 'sweep_vmp': '19.614', 'sweep_pmax': '8.168', 'sweep_voc': '22.003', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875704415 ms)
2025-02-17 23:11:24.094 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+875761914 ms)
2025-02-17 23:11:24.095 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+875761914 ms)
2025-02-17 23:11:24.095 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42386)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+875761914 ms)
2025-02-17 23:11:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875764413 ms)
2025-02-17 23:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875764414 ms)
2025-02-17 23:11:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:11:25.400234+00:00', 'runtime': '875760.001', 'adc_vb_f': '13.055', 'adc_va_f': '19.482', 'adc_vl_f': '13.043', 'adc_ic_f': '0.488', 'adc_il_f': '0.367', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.071', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.387', 'sweep_vmp': '19.51', 'sweep_pmax': '7.036', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875764417 ms)
2025-02-17 23:11:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:10:25.400161+00:00', 'runtime': '875700.001', 'adc_vb_f': '13.092', 'adc_va_f': '19.583', 'adc_vl_f': '13.077', 'adc_ic_f': '0.534', 'adc_il_f': '0.365', 't_hs': '20', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.095', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.976', 'sweep_vmp': '19.614', 'sweep_pmax': '8.168', 'sweep_voc': '22.003', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875764417 ms)
2025-02-17 23:12:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875824415 ms)
2025-02-17 23:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875824415 ms)
2025-02-17 23:12:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:12:25.400186+00:00', 'runtime': '875820.001', 'adc_vb_f': '13.052', 'adc_va_f': '19.485', 'adc_vl_f': '13.004', 'adc_ic_f': '0.425', 'adc_il_f': '0.374', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.052', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.481', 'sweep_vmp': '19.51', 'sweep_pmax': '7.036', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875824419 ms)
2025-02-17 23:12:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:11:25.400234+00:00', 'runtime': '875760.001', 'adc_vb_f': '13.055', 'adc_va_f': '19.482', 'adc_vl_f': '13.043', 'adc_ic_f': '0.488', 'adc_il_f': '0.367', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.071', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.387', 'sweep_vmp': '19.51', 'sweep_pmax': '7.036', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875824420 ms)
2025-02-17 23:13:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875884414 ms)
2025-02-17 23:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875884415 ms)
2025-02-17 23:13:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:13:25.400182+00:00', 'runtime': '875880.001', 'adc_vb_f': '13.022', 'adc_va_f': '19.479', 'adc_vl_f': '12.994', 'adc_ic_f': '0.374', 'adc_il_f': '0.358', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.034', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.862', 'sweep_vmp': '19.51', 'sweep_pmax': '7.036', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875884419 ms)
2025-02-17 23:13:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:12:25.400186+00:00', 'runtime': '875820.001', 'adc_vb_f': '13.052', 'adc_va_f': '19.485', 'adc_vl_f': '13.004', 'adc_ic_f': '0.425', 'adc_il_f': '0.374', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.052', 'vb_ref': '13.971', 'ahc_r': '2660.3', 'ahc_t': '2660.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.481', 'sweep_vmp': '19.51', 'sweep_pmax': '7.036', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.3', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875884419 ms)
2025-02-17 23:14:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+875944415 ms)
2025-02-17 23:14:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+875944416 ms)
2025-02-17 23:14:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:14:25.400210+00:00', 'runtime': '875940.001', 'adc_vb_f': '13.016', 'adc_va_f': '19.479', 'adc_vl_f': '12.967', 'adc_ic_f': '0.374', 'adc_il_f': '0.365', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.016', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.832', 'sweep_vmp': '19.516', 'sweep_pmax': '5.013', 'sweep_voc': '21.765', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875944420 ms)
2025-02-17 23:14:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:13:25.400182+00:00', 'runtime': '875880.001', 'adc_vb_f': '13.022', 'adc_va_f': '19.479', 'adc_vl_f': '12.994', 'adc_ic_f': '0.374', 'adc_il_f': '0.358', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.034', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.862', 'sweep_vmp': '19.51', 'sweep_pmax': '7.036', 'sweep_voc': '21.875', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+875944420 ms)
2025-02-17 23:15:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876004415 ms)
2025-02-17 23:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876004415 ms)
2025-02-17 23:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:15:25.400167+00:00', 'runtime': '876000.001', 'adc_vb_f': '13.013', 'adc_va_f': '19.482', 'adc_vl_f': '12.967', 'adc_ic_f': '0.372', 'adc_il_f': '0.319', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.013', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.771', 'sweep_vmp': '19.516', 'sweep_pmax': '5.013', 'sweep_voc': '21.765', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876004419 ms)
2025-02-17 23:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:14:25.400210+00:00', 'runtime': '875940.001', 'adc_vb_f': '13.016', 'adc_va_f': '19.479', 'adc_vl_f': '12.967', 'adc_ic_f': '0.374', 'adc_il_f': '0.365', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.016', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.832', 'sweep_vmp': '19.516', 'sweep_pmax': '5.013', 'sweep_voc': '21.765', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876004420 ms)
2025-02-17 23:16:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876064414 ms)
2025-02-17 23:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876064415 ms)
2025-02-17 23:16:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:16:25.400228+00:00', 'runtime': '876060.001', 'adc_vb_f': '12.976', 'adc_va_f': '19.482', 'adc_vl_f': '12.927', 'adc_ic_f': '0.278', 'adc_il_f': '0.353', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.982', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.593', 'sweep_vmp': '19.516', 'sweep_pmax': '5.013', 'sweep_voc': '21.765', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876064417 ms)
2025-02-17 23:16:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:15:25.400167+00:00', 'runtime': '876000.001', 'adc_vb_f': '13.013', 'adc_va_f': '19.482', 'adc_vl_f': '12.967', 'adc_ic_f': '0.372', 'adc_il_f': '0.319', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.013', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.771', 'sweep_vmp': '19.516', 'sweep_pmax': '5.013', 'sweep_voc': '21.765', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876064417 ms)
2025-02-17 23:16:34.559 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+876072379 ms)
2025-02-17 23:16:34.560 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+876072379 ms)
2025-02-17 23:16:34.560 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42722)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+876072380 ms)
2025-02-17 23:17:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876124416 ms)
2025-02-17 23:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876124416 ms)
2025-02-17 23:17:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:17:25.400199+00:00', 'runtime': '876120.001', 'adc_vb_f': '12.939', 'adc_va_f': '15.765', 'adc_vl_f': '12.897', 'adc_ic_f': '0.155', 'adc_il_f': '0.362', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.955', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.008', 'sweep_vmp': '15.768', 'sweep_pmax': '5.013', 'sweep_voc': '21.021', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876124420 ms)
2025-02-17 23:17:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:16:25.400228+00:00', 'runtime': '876060.001', 'adc_vb_f': '12.976', 'adc_va_f': '19.482', 'adc_vl_f': '12.927', 'adc_ic_f': '0.278', 'adc_il_f': '0.353', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.982', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.593', 'sweep_vmp': '19.516', 'sweep_pmax': '5.013', 'sweep_voc': '21.765', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876124420 ms)
2025-02-17 23:17:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876133435 ms)
2025-02-17 23:17:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876133435 ms)
2025-02-17 23:17:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:17:25.400345+00:00'} (T+876133436 ms)
2025-02-17 23:18:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876184417 ms)
2025-02-17 23:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876184418 ms)
2025-02-17 23:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:18:25.400267+00:00', 'runtime': '876180.001', 'adc_vb_f': '12.915', 'adc_va_f': '15.378', 'adc_vl_f': '12.891', 'adc_ic_f': '0.135', 'adc_il_f': '0.348', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.933', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.691', 'sweep_vmp': '15.366', 'sweep_pmax': '5.013', 'sweep_voc': '20.483', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876184420 ms)
2025-02-17 23:18:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:17:25.400199+00:00', 'runtime': '876120.001', 'adc_vb_f': '12.939', 'adc_va_f': '15.765', 'adc_vl_f': '12.897', 'adc_ic_f': '0.155', 'adc_il_f': '0.362', 't_hs': '19', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.955', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.008', 'sweep_vmp': '15.768', 'sweep_pmax': '5.013', 'sweep_voc': '21.021', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876184420 ms)
2025-02-17 23:18:32.772 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+876190591 ms)
2025-02-17 23:18:32.773 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+876190592 ms)
2025-02-17 23:18:32.773 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42848)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+876190593 ms)
2025-02-17 23:18:35.953 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876193773 ms)
2025-02-17 23:18:35.954 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876193773 ms)
2025-02-17 23:18:35.955 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:18:25.400268+00:00'} (T+876193775 ms)
2025-02-17 23:19:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876244419 ms)
2025-02-17 23:19:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876244419 ms)
2025-02-17 23:19:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:19:25.400149+00:00', 'runtime': '876240.001', 'adc_vb_f': '12.9', 'adc_va_f': '15.378', 'adc_vl_f': '12.891', 'adc_ic_f': '0.13', 'adc_il_f': '0.321', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.903', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.631', 'sweep_vmp': '15.353', 'sweep_pmax': '5.013', 'sweep_voc': '20.468', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876244423 ms)
2025-02-17 23:19:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:18:25.400267+00:00', 'runtime': '876180.001', 'adc_vb_f': '12.915', 'adc_va_f': '15.378', 'adc_vl_f': '12.891', 'adc_ic_f': '0.135', 'adc_il_f': '0.348', 't_hs': '18', 't_batt': '20', 't_amb': '20', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.933', 'vb_ref': '13.971', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.691', 'sweep_vmp': '15.366', 'sweep_pmax': '5.013', 'sweep_voc': '20.483', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876244423 ms)
2025-02-17 23:20:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876304404 ms)
2025-02-17 23:20:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876304405 ms)
2025-02-17 23:20:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:20:25.400158+00:00', 'runtime': '876300.001', 'adc_vb_f': '12.9', 'adc_va_f': '15.228', 'adc_vl_f': '12.854', 'adc_ic_f': '0.114', 'adc_il_f': '0.353', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.495', 'sweep_vmp': '15.216', 'sweep_pmax': '5.013', 'sweep_voc': '20.285', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876304409 ms)
2025-02-17 23:20:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:19:25.400149+00:00', 'runtime': '876240.001', 'adc_vb_f': '12.9', 'adc_va_f': '15.378', 'adc_vl_f': '12.891', 'adc_ic_f': '0.13', 'adc_il_f': '0.321', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.903', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.631', 'sweep_vmp': '15.353', 'sweep_pmax': '5.013', 'sweep_voc': '20.468', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876304409 ms)
2025-02-17 23:21:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876364405 ms)
2025-02-17 23:21:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876364405 ms)
2025-02-17 23:21:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:21:25.400259+00:00', 'runtime': '876360.001', 'adc_vb_f': '12.9', 'adc_va_f': '15.186', 'adc_vl_f': '12.851', 'adc_ic_f': '0.104', 'adc_il_f': '0.37', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.359', 'sweep_vmp': '15.152', 'sweep_pmax': '5.013', 'sweep_voc': '20.2', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876364409 ms)
2025-02-17 23:21:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:20:25.400158+00:00', 'runtime': '876300.001', 'adc_vb_f': '12.9', 'adc_va_f': '15.228', 'adc_vl_f': '12.854', 'adc_ic_f': '0.114', 'adc_il_f': '0.353', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.495', 'sweep_vmp': '15.216', 'sweep_pmax': '5.013', 'sweep_voc': '20.285', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876364410 ms)
2025-02-17 23:22:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876424405 ms)
2025-02-17 23:22:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876424406 ms)
2025-02-17 23:22:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:22:25.400192+00:00', 'runtime': '876420.001', 'adc_vb_f': '12.875', 'adc_va_f': '15.143', 'adc_vl_f': '12.851', 'adc_ic_f': '0.094', 'adc_il_f': '0.372', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.888', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.208', 'sweep_vmp': '15.118', 'sweep_pmax': '5.013', 'sweep_voc': '20.154', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876424410 ms)
2025-02-17 23:22:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:21:25.400259+00:00', 'runtime': '876360.001', 'adc_vb_f': '12.9', 'adc_va_f': '15.186', 'adc_vl_f': '12.851', 'adc_ic_f': '0.104', 'adc_il_f': '0.37', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1592.9', 'ahl_t': '1592.9', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.359', 'sweep_vmp': '15.152', 'sweep_pmax': '5.013', 'sweep_voc': '20.2', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876424411 ms)
2025-02-17 23:22:35.741 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876433560 ms)
2025-02-17 23:22:35.741 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876433561 ms)
2025-02-17 23:22:35.742 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:22:25.400350+00:00'} (T+876433562 ms)
2025-02-17 23:23:16.221 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+876474040 ms)
2025-02-17 23:23:16.221 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+876474041 ms)
2025-02-17 23:23:16.222 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43148)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+876474041 ms)
2025-02-17 23:23:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876484406 ms)
2025-02-17 23:23:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876484406 ms)
2025-02-17 23:23:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:23:25.400198+00:00', 'runtime': '876480.001', 'adc_vb_f': '12.863', 'adc_va_f': '15.085', 'adc_vl_f': '12.848', 'adc_ic_f': '0.082', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.869', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.042', 'sweep_vmp': '15.045', 'sweep_pmax': '5.013', 'sweep_voc': '20.056', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876484409 ms)
2025-02-17 23:23:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:22:25.400192+00:00', 'runtime': '876420.001', 'adc_vb_f': '12.875', 'adc_va_f': '15.143', 'adc_vl_f': '12.851', 'adc_ic_f': '0.094', 'adc_il_f': '0.372', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.888', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.208', 'sweep_vmp': '15.118', 'sweep_pmax': '5.013', 'sweep_voc': '20.154', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876484410 ms)
2025-02-17 23:24:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876544407 ms)
2025-02-17 23:24:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876544408 ms)
2025-02-17 23:24:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:24:25.400208+00:00', 'runtime': '876540.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.984', 'adc_vl_f': '12.817', 'adc_ic_f': '0.072', 'adc_il_f': '0.382', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.921', 'sweep_vmp': '14.981', 'sweep_pmax': '5.013', 'sweep_voc': '19.971', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876544412 ms)
2025-02-17 23:24:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:23:25.400198+00:00', 'runtime': '876480.001', 'adc_vb_f': '12.863', 'adc_va_f': '15.085', 'adc_vl_f': '12.848', 'adc_ic_f': '0.082', 'adc_il_f': '0.362', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.869', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.042', 'sweep_vmp': '15.045', 'sweep_pmax': '5.013', 'sweep_voc': '20.056', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876544412 ms)
2025-02-17 23:25:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876604409 ms)
2025-02-17 23:25:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876604409 ms)
2025-02-17 23:25:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:25:25.400168+00:00', 'runtime': '876600.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.987', 'adc_vl_f': '12.817', 'adc_ic_f': '0.063', 'adc_il_f': '0.331', 't_hs': '18', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.77', 'sweep_vmp': '14.963', 'sweep_pmax': '5.013', 'sweep_voc': '19.946', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876604413 ms)
2025-02-17 23:25:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:24:25.400208+00:00', 'runtime': '876540.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.984', 'adc_vl_f': '12.817', 'adc_ic_f': '0.072', 'adc_il_f': '0.382', 't_hs': '18', 't_batt': '19', 't_amb': '19', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.001', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.921', 'sweep_vmp': '14.981', 'sweep_pmax': '5.013', 'sweep_voc': '19.971', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876604414 ms)
2025-02-17 23:26:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876664407 ms)
2025-02-17 23:26:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876664408 ms)
2025-02-17 23:26:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:26:25.400610+00:00', 'runtime': '876660.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.893', 'adc_vl_f': '12.814', 'adc_ic_f': '0.053', 'adc_il_f': '0.355', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.679', 'sweep_vmp': '14.88', 'sweep_pmax': '5.013', 'sweep_voc': '19.836', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876664410 ms)
2025-02-17 23:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:25:25.400168+00:00', 'runtime': '876600.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.987', 'adc_vl_f': '12.817', 'adc_ic_f': '0.063', 'adc_il_f': '0.331', 't_hs': '18', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.77', 'sweep_vmp': '14.963', 'sweep_pmax': '5.013', 'sweep_voc': '19.946', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876664410 ms)
2025-02-17 23:27:24.598 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+876722417 ms)
2025-02-17 23:27:24.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+876722417 ms)
2025-02-17 23:27:24.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43420)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+876722418 ms)
2025-02-17 23:27:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876724407 ms)
2025-02-17 23:27:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876724408 ms)
2025-02-17 23:27:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:27:25.400604+00:00', 'runtime': '876720.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.844', 'adc_vl_f': '12.814', 'adc_ic_f': '0.043', 'adc_il_f': '0.316', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.86', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.559', 'sweep_vmp': '14.798', 'sweep_pmax': '5.013', 'sweep_voc': '19.727', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876724411 ms)
2025-02-17 23:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:26:25.400610+00:00', 'runtime': '876660.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.893', 'adc_vl_f': '12.814', 'adc_ic_f': '0.053', 'adc_il_f': '0.355', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.679', 'sweep_vmp': '14.88', 'sweep_pmax': '5.013', 'sweep_voc': '19.836', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876724411 ms)
2025-02-17 23:27:56.713 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+876754532 ms)
2025-02-17 23:27:56.713 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+876754532 ms)
2025-02-17 23:27:56.714 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43456)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+876754533 ms)
2025-02-17 23:28:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876784423 ms)
2025-02-17 23:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876784424 ms)
2025-02-17 23:28:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:28:25.400204+00:00', 'runtime': '876780.001', 'adc_vb_f': '12.833', 'adc_va_f': '14.749', 'adc_vl_f': '12.814', 'adc_ic_f': '0.036', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.839', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.423', 'sweep_vmp': '14.716', 'sweep_pmax': '5.013', 'sweep_voc': '19.617', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876784427 ms)
2025-02-17 23:28:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:27:25.400604+00:00', 'runtime': '876720.001', 'adc_vb_f': '12.863', 'adc_va_f': '14.844', 'adc_vl_f': '12.814', 'adc_ic_f': '0.043', 'adc_il_f': '0.316', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.86', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.559', 'sweep_vmp': '14.798', 'sweep_pmax': '5.013', 'sweep_voc': '19.727', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876784427 ms)
2025-02-17 23:29:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876844407 ms)
2025-02-17 23:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876844407 ms)
2025-02-17 23:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:29:25.400799+00:00', 'runtime': '876840.001', 'adc_vb_f': '12.823', 'adc_va_f': '14.651', 'adc_vl_f': '12.814', 'adc_ic_f': '0.029', 'adc_il_f': '0.312', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.827', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.347', 'sweep_vmp': '14.627', 'sweep_pmax': '5.013', 'sweep_voc': '19.501', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876844411 ms)
2025-02-17 23:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:28:25.400204+00:00', 'runtime': '876780.001', 'adc_vb_f': '12.833', 'adc_va_f': '14.749', 'adc_vl_f': '12.814', 'adc_ic_f': '0.036', 'adc_il_f': '0.348', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.839', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.423', 'sweep_vmp': '14.716', 'sweep_pmax': '5.013', 'sweep_voc': '19.617', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876844412 ms)
2025-02-17 23:30:18.958 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+876896777 ms)
2025-02-17 23:30:18.958 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+876896778 ms)
2025-02-17 23:30:18.959 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43606)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+876896778 ms)
2025-02-17 23:30:26.022 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+876903841 ms)
2025-02-17 23:30:26.022 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+876903842 ms)
2025-02-17 23:30:26.023 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43614)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+876903842 ms)
2025-02-17 23:30:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876904423 ms)
2025-02-17 23:30:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876904423 ms)
2025-02-17 23:30:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:30:25.400974+00:00', 'runtime': '876900.001', 'adc_vb_f': '12.823', 'adc_va_f': '14.551', 'adc_vl_f': '12.808', 'adc_ic_f': '0.022', 'adc_il_f': '0.391', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.272', 'sweep_vmp': '14.545', 'sweep_pmax': '5.013', 'sweep_voc': '19.391', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876904426 ms)
2025-02-17 23:30:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:29:25.400799+00:00', 'runtime': '876840.001', 'adc_vb_f': '12.823', 'adc_va_f': '14.651', 'adc_vl_f': '12.814', 'adc_ic_f': '0.029', 'adc_il_f': '0.312', 't_hs': '17', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.827', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.347', 'sweep_vmp': '14.627', 'sweep_pmax': '5.013', 'sweep_voc': '19.501', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876904426 ms)
2025-02-17 23:31:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+876964423 ms)
2025-02-17 23:31:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+876964423 ms)
2025-02-17 23:31:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:31:25.402782+00:00', 'runtime': '876960.003', 'adc_vb_f': '12.823', 'adc_va_f': '14.45', 'adc_vl_f': '12.778', 'adc_ic_f': '0.014', 'adc_il_f': '0.44', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.181', 'sweep_vmp': '14.407', 'sweep_pmax': '5.013', 'sweep_voc': '19.208', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876964427 ms)
2025-02-17 23:31:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:30:25.400974+00:00', 'runtime': '876900.001', 'adc_vb_f': '12.823', 'adc_va_f': '14.551', 'adc_vl_f': '12.808', 'adc_ic_f': '0.022', 'adc_il_f': '0.391', 't_hs': '16', 't_batt': '18', 't_amb': '18', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.03', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.272', 'sweep_vmp': '14.545', 'sweep_pmax': '5.013', 'sweep_voc': '19.391', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+876964428 ms)
2025-02-17 23:31:35.827 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876973646 ms)
2025-02-17 23:31:35.827 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+876973647 ms)
2025-02-17 23:31:35.828 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:31:25.400125+00:00'} (T+876973648 ms)
2025-02-17 23:32:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877024422 ms)
2025-02-17 23:32:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877024423 ms)
2025-02-17 23:32:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:32:25.402621+00:00', 'runtime': '877020.003', 'adc_vb_f': '12.823', 'adc_va_f': '14.355', 'adc_vl_f': '12.775', 'adc_ic_f': '0.012', 'adc_il_f': '0.466', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '14.322', 'sweep_pmax': '5.013', 'sweep_voc': '19.092', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877024427 ms)
2025-02-17 23:32:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:31:25.402782+00:00', 'runtime': '876960.003', 'adc_vb_f': '12.823', 'adc_va_f': '14.45', 'adc_vl_f': '12.778', 'adc_ic_f': '0.014', 'adc_il_f': '0.44', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.181', 'sweep_vmp': '14.407', 'sweep_pmax': '5.013', 'sweep_voc': '19.208', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877024427 ms)
2025-02-17 23:32:35.267 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+877033086 ms)
2025-02-17 23:32:35.267 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+877033086 ms)
2025-02-17 23:32:35.268 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43752)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877033087 ms)
2025-02-17 23:32:35.691 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877033510 ms)
2025-02-17 23:32:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877033511 ms)
2025-02-17 23:32:35.693 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:32:25.399998+00:00'} (T+877033512 ms)
2025-02-17 23:33:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877084405 ms)
2025-02-17 23:33:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877084406 ms)
2025-02-17 23:33:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:33:25.400172+00:00', 'runtime': '877080.001', 'adc_vb_f': '12.823', 'adc_va_f': '14.212', 'adc_vl_f': '12.775', 'adc_ic_f': '0.007', 'adc_il_f': '0.437', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '14.2', 'sweep_pmax': '5.013', 'sweep_voc': '18.93', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877084410 ms)
2025-02-17 23:33:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:32:25.402621+00:00', 'runtime': '877020.003', 'adc_vb_f': '12.823', 'adc_va_f': '14.355', 'adc_vl_f': '12.775', 'adc_ic_f': '0.012', 'adc_il_f': '0.466', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '14.322', 'sweep_pmax': '5.013', 'sweep_voc': '19.092', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877084410 ms)
2025-02-17 23:33:35.860 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877093680 ms)
2025-02-17 23:33:35.861 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877093680 ms)
2025-02-17 23:33:35.866 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:33:25.400137+00:00'} (T+877093685 ms)
2025-02-17 23:34:25.423 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
 (T+877143242 ms)
2025-02-17 23:34:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877144421 ms)
2025-02-17 23:34:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877144422 ms)
2025-02-17 23:34:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:34:25.400515+00:00', 'runtime': '877140.001', 'adc_vb_f': '12.82', 'adc_va_f': '14.111', 'adc_vl_f': '12.775', 'adc_ic_f': '0.007', 'adc_il_f': '0.437', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.82', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '14.075', 'sweep_pmax': '5.013', 'sweep_voc': '18.762', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877144425 ms)
2025-02-17 23:34:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:33:25.400172+00:00', 'runtime': '877080.001', 'adc_vb_f': '12.823', 'adc_va_f': '14.212', 'adc_vl_f': '12.775', 'adc_ic_f': '0.007', 'adc_il_f': '0.437', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.823', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '14.2', 'sweep_pmax': '5.013', 'sweep_voc': '18.93', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877144426 ms)
2025-02-17 23:34:59.526 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+877177346 ms)
2025-02-17 23:34:59.527 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+877177346 ms)
2025-02-17 23:34:59.527 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43906)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877177346 ms)
2025-02-17 23:35:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877204407 ms)
2025-02-17 23:35:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877204408 ms)
2025-02-17 23:35:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:35:25.400169+00:00', 'runtime': '877200.001', 'adc_vb_f': '12.802', 'adc_va_f': '13.962', 'adc_vl_f': '12.775', 'adc_ic_f': '0.007', 'adc_il_f': '0.396', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.811', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '13.919', 'sweep_pmax': '5.013', 'sweep_voc': '18.555', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877204411 ms)
2025-02-17 23:35:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:34:25.400515+00:00', 'runtime': '877140.001', 'adc_vb_f': '12.82', 'adc_va_f': '14.111', 'adc_vl_f': '12.775', 'adc_ic_f': '0.007', 'adc_il_f': '0.437', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.82', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '14.075', 'sweep_pmax': '5.013', 'sweep_voc': '18.762', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877204411 ms)
2025-02-17 23:36:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877264404 ms)
2025-02-17 23:36:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877264405 ms)
2025-02-17 23:36:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:36:25.400191+00:00', 'runtime': '877260.001', 'adc_vb_f': '12.796', 'adc_va_f': '13.77', 'adc_vl_f': '12.775', 'adc_ic_f': '0.017', 'adc_il_f': '0.401', 't_hs': '15', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.796', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.211', 'sweep_vmp': '13.754', 'sweep_pmax': '5.013', 'sweep_voc': '18.335', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877264406 ms)
2025-02-17 23:36:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:35:25.400169+00:00', 'runtime': '877200.001', 'adc_vb_f': '12.802', 'adc_va_f': '13.962', 'adc_vl_f': '12.775', 'adc_ic_f': '0.007', 'adc_il_f': '0.396', 't_hs': '16', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.811', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '13.919', 'sweep_pmax': '5.013', 'sweep_voc': '18.555', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877264406 ms)
2025-02-17 23:37:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877324420 ms)
2025-02-17 23:37:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877324421 ms)
2025-02-17 23:37:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:37:25.400166+00:00', 'runtime': '877320.001', 'adc_vb_f': '12.793', 'adc_va_f': '13.513', 'adc_vl_f': '12.775', 'adc_ic_f': '0.024', 'adc_il_f': '0.406', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.793', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.302', 'sweep_vmp': '13.516', 'sweep_pmax': '5.013', 'sweep_voc': '18.018', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877324422 ms)
2025-02-17 23:37:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:36:25.400191+00:00', 'runtime': '877260.001', 'adc_vb_f': '12.796', 'adc_va_f': '13.77', 'adc_vl_f': '12.775', 'adc_ic_f': '0.017', 'adc_il_f': '0.401', 't_hs': '15', 't_batt': '17', 't_amb': '17', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.796', 'vb_ref': '14.06', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.0', 'ahl_t': '1593.0', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.211', 'sweep_vmp': '13.754', 'sweep_pmax': '5.013', 'sweep_voc': '18.335', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877324422 ms)
2025-02-17 23:38:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877384404 ms)
2025-02-17 23:38:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877384404 ms)
2025-02-17 23:38:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:38:25.400174+00:00', 'runtime': '877380.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.513', 'adc_vl_f': '12.775', 'adc_ic_f': '0.019', 'adc_il_f': '0.408', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.242', 'sweep_vmp': '13.269', 'sweep_pmax': '5.013', 'sweep_voc': '17.688', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877384405 ms)
2025-02-17 23:38:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:37:25.400166+00:00', 'runtime': '877320.001', 'adc_vb_f': '12.793', 'adc_va_f': '13.513', 'adc_vl_f': '12.775', 'adc_ic_f': '0.024', 'adc_il_f': '0.406', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.793', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.302', 'sweep_vmp': '13.516', 'sweep_pmax': '5.013', 'sweep_voc': '18.018', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877384406 ms)
2025-02-17 23:39:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877444421 ms)
2025-02-17 23:39:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877444422 ms)
2025-02-17 23:39:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:39:25.400201+00:00', 'runtime': '877440.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.513', 'adc_vl_f': '12.775', 'adc_ic_f': '0.019', 'adc_il_f': '0.428', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.211', 'sweep_vmp': '12.976', 'sweep_pmax': '5.013', 'sweep_voc': '17.297', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877444423 ms)
2025-02-17 23:39:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:38:25.400174+00:00', 'runtime': '877380.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.513', 'adc_vl_f': '12.775', 'adc_ic_f': '0.019', 'adc_il_f': '0.408', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.242', 'sweep_vmp': '13.269', 'sweep_pmax': '5.013', 'sweep_voc': '17.688', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877444423 ms)
2025-02-17 23:39:36.379 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877454198 ms)
2025-02-17 23:39:36.380 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877454199 ms)
2025-02-17 23:39:36.381 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:39:25.405062+00:00'} (T+877454200 ms)
2025-02-17 23:39:37.881 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+877455701 ms)
2025-02-17 23:39:37.882 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+877455701 ms)
2025-02-17 23:39:37.882 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877455701 ms)
2025-02-17 23:40:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877504421 ms)
2025-02-17 23:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877504421 ms)
2025-02-17 23:40:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:40:25.400148+00:00', 'runtime': '877500.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.48', 'adc_vl_f': '12.775', 'adc_ic_f': '0.017', 'adc_il_f': '0.452', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.181', 'sweep_vmp': '12.711', 'sweep_pmax': '5.013', 'sweep_voc': '16.943', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877504425 ms)
2025-02-17 23:40:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:39:25.400201+00:00', 'runtime': '877440.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.513', 'adc_vl_f': '12.775', 'adc_ic_f': '0.019', 'adc_il_f': '0.428', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.211', 'sweep_vmp': '12.976', 'sweep_pmax': '5.013', 'sweep_voc': '17.297', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877504426 ms)
2025-02-17 23:40:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877513438 ms)
2025-02-17 23:40:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877513438 ms)
2025-02-17 23:40:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:40:25.405138+00:00'} (T+877513439 ms)
2025-02-17 23:40:42.031 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+877519851 ms)
2025-02-17 23:40:42.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+877519851 ms)
2025-02-17 23:40:42.032 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44276)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877519851 ms)
2025-02-17 23:41:03.131 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+877540951 ms)
2025-02-17 23:41:03.132 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+877540951 ms)
2025-02-17 23:41:03.132 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44300)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877540951 ms)
2025-02-17 23:41:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877564417 ms)
2025-02-17 23:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877564418 ms)
2025-02-17 23:41:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:41:25.400130+00:00', 'runtime': '877560.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.428', 'adc_vl_f': '12.775', 'adc_ic_f': '0.014', 'adc_il_f': '0.459', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.151', 'sweep_vmp': '12.308', 'sweep_pmax': '5.013', 'sweep_voc': '16.406', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877564421 ms)
2025-02-17 23:41:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:40:25.400148+00:00', 'runtime': '877500.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.48', 'adc_vl_f': '12.775', 'adc_ic_f': '0.017', 'adc_il_f': '0.452', 't_hs': '15', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.181', 'sweep_vmp': '12.711', 'sweep_pmax': '5.013', 'sweep_voc': '16.943', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877564421 ms)
2025-02-17 23:41:35.732 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877573552 ms)
2025-02-17 23:41:35.733 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877573552 ms)
2025-02-17 23:41:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:41:25.405130+00:00'} (T+877573553 ms)
2025-02-17 23:42:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877624417 ms)
2025-02-17 23:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877624417 ms)
2025-02-17 23:42:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:42:25.400064+00:00', 'runtime': '877620.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.406', 'adc_vl_f': '12.775', 'adc_ic_f': '0.012', 'adc_il_f': '0.437', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '11.835', 'sweep_pmax': '5.013', 'sweep_voc': '15.778', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877624421 ms)
2025-02-17 23:42:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:41:25.400130+00:00', 'runtime': '877560.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.428', 'adc_vl_f': '12.775', 'adc_ic_f': '0.014', 'adc_il_f': '0.459', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.151', 'sweep_vmp': '12.308', 'sweep_pmax': '5.013', 'sweep_voc': '16.406', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877624422 ms)
2025-02-17 23:42:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877633428 ms)
2025-02-17 23:42:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877633429 ms)
2025-02-17 23:42:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:42:25.405127+00:00'} (T+877633433 ms)
2025-02-17 23:42:54.382 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+877652201 ms)
2025-02-17 23:42:54.382 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+877652201 ms)
2025-02-17 23:42:54.383 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44420)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877652202 ms)
2025-02-17 23:43:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877684414 ms)
2025-02-17 23:43:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877684414 ms)
2025-02-17 23:43:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:43:25.400177+00:00', 'runtime': '877680.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.406', 'adc_vl_f': '12.775', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '11.337', 'sweep_pmax': '5.013', 'sweep_voc': '15.112', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877684416 ms)
2025-02-17 23:43:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:42:25.400064+00:00', 'runtime': '877620.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.406', 'adc_vl_f': '12.775', 'adc_ic_f': '0.012', 'adc_il_f': '0.437', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '11.835', 'sweep_pmax': '5.013', 'sweep_voc': '15.778', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877684417 ms)
2025-02-17 23:43:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877693432 ms)
2025-02-17 23:43:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877693432 ms)
2025-02-17 23:43:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:43:25.400194+00:00'} (T+877693433 ms)
2025-02-17 23:44:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877744411 ms)
2025-02-17 23:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877744412 ms)
2025-02-17 23:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:44:25.400205+00:00', 'runtime': '877740.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.406', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.416', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '10.861', 'sweep_pmax': '5.013', 'sweep_voc': '14.478', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877744416 ms)
2025-02-17 23:44:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:43:25.400177+00:00', 'runtime': '877680.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.406', 'adc_vl_f': '12.775', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '14', 't_batt': '16', 't_amb': '16', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.089', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '11.337', 'sweep_pmax': '5.013', 'sweep_voc': '15.112', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877744416 ms)
2025-02-17 23:44:35.729 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877753548 ms)
2025-02-17 23:44:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877753549 ms)
2025-02-17 23:44:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:44:25.400197+00:00'} (T+877753550 ms)
2025-02-17 23:45:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877804422 ms)
2025-02-17 23:45:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877804423 ms)
2025-02-17 23:45:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:45:25.400177+00:00', 'runtime': '877800.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.388', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.413', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.458', 'sweep_pmax': '5.013', 'sweep_voc': '13.94', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877804425 ms)
2025-02-17 23:45:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:44:25.400205+00:00', 'runtime': '877740.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.406', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.416', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '10.861', 'sweep_pmax': '5.013', 'sweep_voc': '14.478', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877804425 ms)
2025-02-17 23:45:36.109 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877813928 ms)
2025-02-17 23:45:36.109 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877813928 ms)
2025-02-17 23:45:36.110 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:45:25.400405+00:00'} (T+877813929 ms)
2025-02-17 23:45:51.647 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+877829466 ms)
2025-02-17 23:45:51.647 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+877829467 ms)
2025-02-17 23:45:51.648 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44620)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877829467 ms)
2025-02-17 23:46:16.778 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+877854597 ms)
2025-02-17 23:46:16.778 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+877854597 ms)
2025-02-17 23:46:16.778 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44646)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+877854598 ms)
2025-02-17 23:46:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877864419 ms)
2025-02-17 23:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877864419 ms)
2025-02-17 23:46:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:46:25.400220+00:00', 'runtime': '877860.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.324', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.442', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.458', 'sweep_pmax': '5.013', 'sweep_voc': '13.94', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877864423 ms)
2025-02-17 23:46:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:45:25.400177+00:00', 'runtime': '877800.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.388', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.413', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.458', 'sweep_pmax': '5.013', 'sweep_voc': '13.94', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877864423 ms)
2025-02-17 23:46:35.955 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877873774 ms)
2025-02-17 23:46:35.956 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877873775 ms)
2025-02-17 23:46:35.956 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:46:25.400231+00:00'} (T+877873776 ms)
2025-02-17 23:47:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877924416 ms)
2025-02-17 23:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877924416 ms)
2025-02-17 23:47:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:47:25.400138+00:00', 'runtime': '877920.001', 'adc_vb_f': '12.784', 'adc_va_f': '13.3', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.435', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.784', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.211', 'sweep_pmax': '5.013', 'sweep_voc': '13.611', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877924421 ms)
2025-02-17 23:47:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:46:25.400220+00:00', 'runtime': '877860.001', 'adc_vb_f': '12.787', 'adc_va_f': '13.324', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.442', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.458', 'sweep_pmax': '5.013', 'sweep_voc': '13.94', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877924421 ms)
2025-02-17 23:47:35.629 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877933449 ms)
2025-02-17 23:47:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877933449 ms)
2025-02-17 23:47:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:47:25.400338+00:00'} (T+877933450 ms)
2025-02-17 23:48:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+877984411 ms)
2025-02-17 23:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+877984411 ms)
2025-02-17 23:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:48:25.402845+00:00', 'runtime': '877980.003', 'adc_vb_f': '12.781', 'adc_va_f': '12.231', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.461', 't_hs': '13', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.784', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.979', 'sweep_pmax': '5.013', 'sweep_voc': '13.303', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877984412 ms)
2025-02-17 23:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:47:25.400138+00:00', 'runtime': '877920.001', 'adc_vb_f': '12.784', 'adc_va_f': '13.3', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.435', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.784', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.211', 'sweep_pmax': '5.013', 'sweep_voc': '13.611', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+877984413 ms)
2025-02-17 23:48:35.733 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877993552 ms)
2025-02-17 23:48:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+877993553 ms)
2025-02-17 23:48:35.735 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:48:25.400240+00:00'} (T+877993554 ms)
2025-02-17 23:49:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878044422 ms)
2025-02-17 23:49:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878044423 ms)
2025-02-17 23:49:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:49:25.402737+00:00', 'runtime': '878040.003', 'adc_vb_f': '12.775', 'adc_va_f': '12.012', 'adc_vl_f': '12.775', 'adc_ic_f': '0.0', 'adc_il_f': '0.466', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.778', 'vb_ref': '14.148', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.396', 'sweep_pmax': '5.013', 'sweep_voc': '12.531', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878044424 ms)
2025-02-17 23:49:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:48:25.402845+00:00', 'runtime': '877980.003', 'adc_vb_f': '12.781', 'adc_va_f': '12.231', 'adc_vl_f': '12.775', 'adc_ic_f': '0.002', 'adc_il_f': '0.461', 't_hs': '13', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.784', 'vb_ref': '14.119', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.979', 'sweep_pmax': '5.013', 'sweep_voc': '13.303', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878044425 ms)
2025-02-17 23:49:35.853 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878053672 ms)
2025-02-17 23:49:35.854 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878053673 ms)
2025-02-17 23:49:35.855 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:49:25.400098+00:00'} (T+878053674 ms)
2025-02-17 23:50:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878104419 ms)
2025-02-17 23:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878104419 ms)
2025-02-17 23:50:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:50:25.402799+00:00', 'runtime': '878100.003', 'adc_vb_f': '12.772', 'adc_va_f': '11.368', 'adc_vl_f': '12.772', 'adc_ic_f': '0.0', 'adc_il_f': '0.461', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.775', 'vb_ref': '14.148', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.015', 'sweep_pmax': '5.013', 'sweep_voc': '12.021', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878104423 ms)
2025-02-17 23:50:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:49:25.402737+00:00', 'runtime': '878040.003', 'adc_vb_f': '12.775', 'adc_va_f': '12.012', 'adc_vl_f': '12.775', 'adc_ic_f': '0.0', 'adc_il_f': '0.466', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.778', 'vb_ref': '14.148', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.396', 'sweep_pmax': '5.013', 'sweep_voc': '12.531', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878104423 ms)
2025-02-17 23:51:18.236 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+878156055 ms)
2025-02-17 23:51:18.236 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+878156056 ms)
2025-02-17 23:51:18.237 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44968)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+878156056 ms)
2025-02-17 23:51:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878164417 ms)
2025-02-17 23:51:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878164417 ms)
2025-02-17 23:51:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:51:25.400822+00:00', 'runtime': '878160.001', 'adc_vb_f': '12.766', 'adc_va_f': '10.51', 'adc_vl_f': '12.772', 'adc_ic_f': '0.007', 'adc_il_f': '0.457', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.769', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878164421 ms)
2025-02-17 23:51:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:50:25.402799+00:00', 'runtime': '878100.003', 'adc_vb_f': '12.772', 'adc_va_f': '11.368', 'adc_vl_f': '12.772', 'adc_ic_f': '0.0', 'adc_il_f': '0.461', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.775', 'vb_ref': '14.148', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.1', 'ahl_t': '1593.1', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '9.015', 'sweep_pmax': '5.013', 'sweep_voc': '12.021', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '8.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878164421 ms)
2025-02-17 23:52:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878224412 ms)
2025-02-17 23:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878224413 ms)
2025-02-17 23:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:52:25.400649+00:00', 'runtime': '878220.001', 'adc_vb_f': '12.766', 'adc_va_f': '9.653', 'adc_vl_f': '12.766', 'adc_ic_f': '0.007', 'adc_il_f': '0.432', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.769', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878224417 ms)
2025-02-17 23:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:51:25.400822+00:00', 'runtime': '878160.001', 'adc_vb_f': '12.766', 'adc_va_f': '10.51', 'adc_vl_f': '12.772', 'adc_ic_f': '0.007', 'adc_il_f': '0.457', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.769', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878224417 ms)
2025-02-17 23:53:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878284408 ms)
2025-02-17 23:53:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878284409 ms)
2025-02-17 23:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:53:25.400183+00:00', 'runtime': '878280.001', 'adc_vb_f': '12.766', 'adc_va_f': '8.795', 'adc_vl_f': '12.766', 'adc_ic_f': '0.007', 'adc_il_f': '0.416', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.762', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878284413 ms)
2025-02-17 23:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:52:25.400649+00:00', 'runtime': '878220.001', 'adc_vb_f': '12.766', 'adc_va_f': '9.653', 'adc_vl_f': '12.766', 'adc_ic_f': '0.007', 'adc_il_f': '0.432', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.769', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878284413 ms)
2025-02-17 23:53:35.854 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878293673 ms)
2025-02-17 23:53:35.854 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878293674 ms)
2025-02-17 23:53:35.856 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:53:25.405073+00:00'} (T+878293675 ms)
2025-02-17 23:54:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878344421 ms)
2025-02-17 23:54:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878344421 ms)
2025-02-17 23:54:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:54:25.400187+00:00', 'runtime': '878340.001', 'adc_vb_f': '12.762', 'adc_va_f': '7.935', 'adc_vl_f': '12.766', 'adc_ic_f': '0.007', 'adc_il_f': '0.42', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.759', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878344423 ms)
2025-02-17 23:54:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:53:25.400183+00:00', 'runtime': '878280.001', 'adc_vb_f': '12.766', 'adc_va_f': '8.795', 'adc_vl_f': '12.766', 'adc_ic_f': '0.007', 'adc_il_f': '0.416', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.762', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20130', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878344423 ms)
2025-02-17 23:54:35.913 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878353733 ms)
2025-02-17 23:54:35.914 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878353733 ms)
2025-02-17 23:54:35.915 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:54:25.413126+00:00'} (T+878353734 ms)
2025-02-17 23:55:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878404417 ms)
2025-02-17 23:55:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878404417 ms)
2025-02-17 23:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:55:25.400182+00:00', 'runtime': '878400.001', 'adc_vb_f': '12.756', 'adc_va_f': '7.077', 'adc_vl_f': '12.756', 'adc_ic_f': '0.007', 'adc_il_f': '0.411', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.756', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878404418 ms)
2025-02-17 23:55:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:54:25.400187+00:00', 'runtime': '878340.001', 'adc_vb_f': '12.762', 'adc_va_f': '7.935', 'adc_vl_f': '12.766', 'adc_ic_f': '0.007', 'adc_il_f': '0.42', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.759', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878404419 ms)
2025-02-17 23:55:36.140 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878413959 ms)
2025-02-17 23:55:36.141 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878413960 ms)
2025-02-17 23:55:36.142 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-17 23:55:25.409126+00:00'} (T+878413961 ms)
2025-02-17 23:56:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878464413 ms)
2025-02-17 23:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878464413 ms)
2025-02-17 23:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:56:25.402760+00:00', 'runtime': '878460.003', 'adc_vb_f': '12.753', 'adc_va_f': '6.238', 'adc_vl_f': '12.741', 'adc_ic_f': '0.007', 'adc_il_f': '0.416', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.753', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878464415 ms)
2025-02-17 23:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:55:25.400182+00:00', 'runtime': '878400.001', 'adc_vb_f': '12.756', 'adc_va_f': '7.077', 'adc_vl_f': '12.756', 'adc_ic_f': '0.007', 'adc_il_f': '0.411', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.756', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878464416 ms)
2025-02-17 23:56:49.699 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+878487518 ms)
2025-02-17 23:56:49.699 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+878487518 ms)
2025-02-17 23:56:49.700 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45328)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+878487519 ms)
2025-02-17 23:57:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878524411 ms)
2025-02-17 23:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878524412 ms)
2025-02-17 23:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:57:25.400172+00:00', 'runtime': '878520.001', 'adc_vb_f': '12.756', 'adc_va_f': '5.469', 'adc_vl_f': '12.744', 'adc_ic_f': '0.007', 'adc_il_f': '0.396', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.753', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878524413 ms)
2025-02-17 23:57:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:56:25.402760+00:00', 'runtime': '878460.003', 'adc_vb_f': '12.753', 'adc_va_f': '6.238', 'adc_vl_f': '12.741', 'adc_ic_f': '0.007', 'adc_il_f': '0.416', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.753', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878524413 ms)
2025-02-17 23:58:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878584406 ms)
2025-02-17 23:58:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878584406 ms)
2025-02-17 23:58:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:58:25.400183+00:00', 'runtime': '878580.001', 'adc_vb_f': '12.753', 'adc_va_f': '4.825', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.753', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878584407 ms)
2025-02-17 23:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:57:25.400172+00:00', 'runtime': '878520.001', 'adc_vb_f': '12.756', 'adc_va_f': '5.469', 'adc_vl_f': '12.744', 'adc_ic_f': '0.007', 'adc_il_f': '0.396', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.753', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878584408 ms)
2025-02-17 23:58:33.503 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+878591322 ms)
2025-02-17 23:59:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878644403 ms)
2025-02-17 23:59:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878644403 ms)
2025-02-17 23:59:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-17 23:59:25.400807+00:00', 'runtime': '878640.001', 'adc_vb_f': '12.75', 'adc_va_f': '4.181', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878644404 ms)
2025-02-17 23:59:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:58:25.400183+00:00', 'runtime': '878580.001', 'adc_vb_f': '12.753', 'adc_va_f': '4.825', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.753', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878644405 ms)
2025-02-17 23:59:59.985 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+878677804 ms)
2025-02-17 23:59:59.985 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+878677805 ms)
2025-02-17 23:59:59.986 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45528)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+878677805 ms)
2025-02-18 00:00:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878704419 ms)
2025-02-18 00:00:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878704420 ms)
2025-02-18 00:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:00:25.400194+00:00', 'runtime': '878700.001', 'adc_vb_f': '12.75', 'adc_va_f': '3.677', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878704421 ms)
2025-02-18 00:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-17 23:59:25.400807+00:00', 'runtime': '878640.001', 'adc_vb_f': '12.75', 'adc_va_f': '4.181', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878704422 ms)
2025-02-18 00:00:35.496 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878713316 ms)
2025-02-18 00:00:35.497 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878713316 ms)
2025-02-18 00:00:35.498 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:00:25.413046+00:00'} (T+878713317 ms)
2025-02-18 00:01:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878764414 ms)
2025-02-18 00:01:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878764414 ms)
2025-02-18 00:01:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:01:25.402807+00:00', 'runtime': '878760.003', 'adc_vb_f': '12.75', 'adc_va_f': '3.278', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878764416 ms)
2025-02-18 00:01:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:00:25.400194+00:00', 'runtime': '878700.001', 'adc_vb_f': '12.75', 'adc_va_f': '3.677', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878764416 ms)
2025-02-18 00:01:35.819 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878773638 ms)
2025-02-18 00:01:35.820 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878773639 ms)
2025-02-18 00:01:35.821 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:01:25.400183+00:00'} (T+878773640 ms)
2025-02-18 00:02:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878824411 ms)
2025-02-18 00:02:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878824411 ms)
2025-02-18 00:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:02:25.402563+00:00', 'runtime': '878820.003', 'adc_vb_f': '12.75', 'adc_va_f': '2.893', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878824412 ms)
2025-02-18 00:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:01:25.402807+00:00', 'runtime': '878760.003', 'adc_vb_f': '12.75', 'adc_va_f': '3.278', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878824413 ms)
2025-02-18 00:02:35.688 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878833508 ms)
2025-02-18 00:02:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878833508 ms)
2025-02-18 00:02:35.690 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:02:25.399996+00:00'} (T+878833509 ms)
2025-02-18 00:03:17.313 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+878875132 ms)
2025-02-18 00:03:17.313 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+878875133 ms)
2025-02-18 00:03:17.314 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45738)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+878875133 ms)
2025-02-18 00:03:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878884406 ms)
2025-02-18 00:03:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878884406 ms)
2025-02-18 00:03:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:03:25.402781+00:00', 'runtime': '878880.003', 'adc_vb_f': '12.75', 'adc_va_f': '2.579', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '11', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878884407 ms)
2025-02-18 00:03:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:02:25.402563+00:00', 'runtime': '878820.003', 'adc_vb_f': '12.75', 'adc_va_f': '2.893', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878884408 ms)
2025-02-18 00:03:35.840 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878893659 ms)
2025-02-18 00:03:35.841 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+878893660 ms)
2025-02-18 00:03:35.844 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:03:25.400191+00:00'} (T+878893663 ms)
2025-02-18 00:04:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+878944421 ms)
2025-02-18 00:04:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+878944421 ms)
2025-02-18 00:04:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:04:25.400259+00:00', 'runtime': '878940.001', 'adc_vb_f': '12.75', 'adc_va_f': '2.359', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '11', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878944422 ms)
2025-02-18 00:04:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:03:25.402781+00:00', 'runtime': '878880.003', 'adc_vb_f': '12.75', 'adc_va_f': '2.579', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '11', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+878944423 ms)
2025-02-18 00:05:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879004414 ms)
2025-02-18 00:05:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879004414 ms)
2025-02-18 00:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:05:25.400189+00:00', 'runtime': '879000.001', 'adc_vb_f': '12.75', 'adc_va_f': '2.13', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879004415 ms)
2025-02-18 00:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:04:25.400259+00:00', 'runtime': '878940.001', 'adc_vb_f': '12.75', 'adc_va_f': '2.359', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '11', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.2', 'ahl_t': '1593.2', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879004415 ms)
2025-02-18 00:06:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879064410 ms)
2025-02-18 00:06:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879064411 ms)
2025-02-18 00:06:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:06:25.400172+00:00', 'runtime': '879060.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.929', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879064412 ms)
2025-02-18 00:06:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:05:25.400189+00:00', 'runtime': '879000.001', 'adc_vb_f': '12.75', 'adc_va_f': '2.13', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879064412 ms)
2025-02-18 00:07:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879124410 ms)
2025-02-18 00:07:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879124410 ms)
2025-02-18 00:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:07:25.400230+00:00', 'runtime': '879120.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.74', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879124411 ms)
2025-02-18 00:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:06:25.400172+00:00', 'runtime': '879060.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.929', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879124412 ms)
2025-02-18 00:07:36.147 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879133966 ms)
2025-02-18 00:07:36.148 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879133967 ms)
2025-02-18 00:07:36.149 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:07:25.400338+00:00'} (T+879133968 ms)
2025-02-18 00:08:05.749 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+879163568 ms)
2025-02-18 00:08:05.749 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+879163568 ms)
2025-02-18 00:08:05.749 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46054)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+879163569 ms)
2025-02-18 00:08:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879184418 ms)
2025-02-18 00:08:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879184419 ms)
2025-02-18 00:08:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:08:25.400159+00:00', 'runtime': '879180.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.608', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879184420 ms)
2025-02-18 00:08:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:07:25.400230+00:00', 'runtime': '879120.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.74', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879184420 ms)
2025-02-18 00:08:36.033 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879193852 ms)
2025-02-18 00:08:36.034 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879193853 ms)
2025-02-18 00:08:36.035 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:08:25.400143+00:00'} (T+879193854 ms)
2025-02-18 00:09:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879244415 ms)
2025-02-18 00:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879244416 ms)
2025-02-18 00:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:09:25.400179+00:00', 'runtime': '879240.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.501', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879244417 ms)
2025-02-18 00:09:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:08:25.400159+00:00', 'runtime': '879180.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.608', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879244417 ms)
2025-02-18 00:10:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879304410 ms)
2025-02-18 00:10:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879304411 ms)
2025-02-18 00:10:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:10:25.400175+00:00', 'runtime': '879300.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.404', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879304412 ms)
2025-02-18 00:10:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:09:25.400179+00:00', 'runtime': '879240.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.501', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.164', 'vb_max_daily': '14.722', 'ahc_daily': '30.4', 'ahl_daily': '9.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879304412 ms)
2025-02-18 00:11:25.088 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+879362907 ms)
2025-02-18 00:11:25.088 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+879362907 ms)
2025-02-18 00:11:25.089 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46264)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+879362908 ms)
2025-02-18 00:11:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879364406 ms)
2025-02-18 00:11:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879364406 ms)
2025-02-18 00:11:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:11:25.400227+00:00', 'runtime': '879360.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.361', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879364407 ms)
2025-02-18 00:11:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:10:25.400175+00:00', 'runtime': '879300.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.404', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879364407 ms)
2025-02-18 00:12:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879424424 ms)
2025-02-18 00:12:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879424424 ms)
2025-02-18 00:12:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:12:25.400177+00:00', 'runtime': '879420.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.285', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879424425 ms)
2025-02-18 00:12:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:11:25.400227+00:00', 'runtime': '879360.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.361', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879424426 ms)
2025-02-18 00:13:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879484412 ms)
2025-02-18 00:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879484413 ms)
2025-02-18 00:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:13:25.400150+00:00', 'runtime': '879480.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.202', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879484414 ms)
2025-02-18 00:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:12:25.400177+00:00', 'runtime': '879420.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.285', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879484414 ms)
2025-02-18 00:14:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879544406 ms)
2025-02-18 00:14:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879544407 ms)
2025-02-18 00:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:14:25.400213+00:00', 'runtime': '879540.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.178', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879544411 ms)
2025-02-18 00:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:13:25.400150+00:00', 'runtime': '879480.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.202', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879544411 ms)
2025-02-18 00:14:35.669 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879553488 ms)
2025-02-18 00:14:35.669 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879553489 ms)
2025-02-18 00:14:35.670 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:14:25.405063+00:00'} (T+879553490 ms)
2025-02-18 00:14:42.436 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+879560255 ms)
2025-02-18 00:14:42.436 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+879560255 ms)
2025-02-18 00:14:42.436 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46472)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+879560256 ms)
2025-02-18 00:15:09.555 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+879587374 ms)
2025-02-18 00:15:09.555 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+879587375 ms)
2025-02-18 00:15:09.556 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46502)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+879587375 ms)
2025-02-18 00:15:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879604418 ms)
2025-02-18 00:15:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879604418 ms)
2025-02-18 00:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:15:25.400183+00:00', 'runtime': '879600.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.169', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879604419 ms)
2025-02-18 00:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:14:25.400213+00:00', 'runtime': '879540.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.178', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879604420 ms)
2025-02-18 00:15:35.823 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879613642 ms)
2025-02-18 00:15:35.824 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879613643 ms)
2025-02-18 00:15:35.825 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:15:25.400145+00:00'} (T+879613644 ms)
2025-02-18 00:16:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879664413 ms)
2025-02-18 00:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879664413 ms)
2025-02-18 00:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:16:25.400006+00:00', 'runtime': '879660.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.096', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.747', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879664414 ms)
2025-02-18 00:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:15:25.400183+00:00', 'runtime': '879600.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.169', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.75', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879664415 ms)
2025-02-18 00:16:35.888 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879673707 ms)
2025-02-18 00:16:35.889 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879673708 ms)
2025-02-18 00:16:35.890 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:16:25.401135+00:00'} (T+879673709 ms)
2025-02-18 00:17:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879724407 ms)
2025-02-18 00:17:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879724408 ms)
2025-02-18 00:17:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:17:25.402843+00:00', 'runtime': '879720.003', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.747', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879724409 ms)
2025-02-18 00:17:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:16:25.400006+00:00', 'runtime': '879660.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.096', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.747', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879724409 ms)
2025-02-18 00:17:35.730 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879733549 ms)
2025-02-18 00:17:35.731 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879733550 ms)
2025-02-18 00:17:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:17:25.400234+00:00'} (T+879733552 ms)
2025-02-18 00:18:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879784419 ms)
2025-02-18 00:18:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879784420 ms)
2025-02-18 00:18:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:18:25.403010+00:00', 'runtime': '879780.004', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.744', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879784421 ms)
2025-02-18 00:18:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:17:25.402843+00:00', 'runtime': '879720.003', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.747', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879784421 ms)
2025-02-18 00:18:51.881 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+879809700 ms)
2025-02-18 00:18:51.881 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+879809701 ms)
2025-02-18 00:18:51.883 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46752)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+879809702 ms)
2025-02-18 00:18:52.953 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+879810772 ms)
2025-02-18 00:18:52.953 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+879810772 ms)
2025-02-18 00:18:52.954 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46754)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+879810773 ms)
2025-02-18 00:19:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879844401 ms)
2025-02-18 00:19:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879844401 ms)
2025-02-18 00:19:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:19:25.400805+00:00', 'runtime': '879840.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.738', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879844402 ms)
2025-02-18 00:19:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:18:25.403010+00:00', 'runtime': '879780.004', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.3', 'ahl_t': '1593.3', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.744', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879844403 ms)
2025-02-18 00:20:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879904416 ms)
2025-02-18 00:20:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879904416 ms)
2025-02-18 00:20:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:20:25.400202+00:00', 'runtime': '879900.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.738', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879904417 ms)
2025-02-18 00:20:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:19:25.400805+00:00', 'runtime': '879840.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.738', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879904418 ms)
2025-02-18 00:20:35.711 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879913530 ms)
2025-02-18 00:20:35.712 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879913531 ms)
2025-02-18 00:20:35.713 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:20:25.405068+00:00'} (T+879913532 ms)
2025-02-18 00:21:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+879964415 ms)
2025-02-18 00:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+879964416 ms)
2025-02-18 00:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:21:25.402738+00:00', 'runtime': '879960.003', 'adc_vb_f': '12.747', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879964417 ms)
2025-02-18 00:21:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:20:25.400202+00:00', 'runtime': '879900.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.738', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+879964417 ms)
2025-02-18 00:21:35.630 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879973449 ms)
2025-02-18 00:21:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+879973449 ms)
2025-02-18 00:21:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:21:25.400113+00:00'} (T+879973450 ms)
2025-02-18 00:22:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880024412 ms)
2025-02-18 00:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880024412 ms)
2025-02-18 00:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:22:25.402798+00:00', 'runtime': '880020.003', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880024413 ms)
2025-02-18 00:22:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:21:25.402738+00:00', 'runtime': '879960.003', 'adc_vb_f': '12.747', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880024414 ms)
2025-02-18 00:22:36.753 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880034573 ms)
2025-02-18 00:22:36.754 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880034573 ms)
2025-02-18 00:22:36.755 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:22:25.400181+00:00'} (T+880034574 ms)
2025-02-18 00:22:42.336 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+880040156 ms)
2025-02-18 00:22:42.337 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+880040156 ms)
2025-02-18 00:22:42.337 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46996)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+880040157 ms)
2025-02-18 00:23:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880084410 ms)
2025-02-18 00:23:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880084411 ms)
2025-02-18 00:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:23:25.400077+00:00', 'runtime': '880080.001', 'adc_vb_f': '12.747', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880084415 ms)
2025-02-18 00:23:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:22:25.402798+00:00', 'runtime': '880020.003', 'adc_vb_f': '12.75', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880084415 ms)
2025-02-18 00:23:35.843 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880093662 ms)
2025-02-18 00:23:35.844 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880093663 ms)
2025-02-18 00:23:35.845 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:23:25.409119+00:00'} (T+880093664 ms)
2025-02-18 00:24:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880144424 ms)
2025-02-18 00:24:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880144425 ms)
2025-02-18 00:24:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:24:25.403534+00:00', 'runtime': '880140.004', 'adc_vb_f': '12.747', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880144429 ms)
2025-02-18 00:24:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:23:25.400077+00:00', 'runtime': '880080.001', 'adc_vb_f': '12.747', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880144429 ms)
2025-02-18 00:24:35.720 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880153539 ms)
2025-02-18 00:24:35.721 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880153540 ms)
2025-02-18 00:24:35.722 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:24:25.400965+00:00'} (T+880153541 ms)
2025-02-18 00:25:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880204422 ms)
2025-02-18 00:25:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880204422 ms)
2025-02-18 00:25:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:25:25.400080+00:00', 'runtime': '880200.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.068', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.744', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880204424 ms)
2025-02-18 00:25:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:24:25.403534+00:00', 'runtime': '880140.004', 'adc_vb_f': '12.747', 'adc_va_f': '1.071', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880204424 ms)
2025-02-18 00:26:08.665 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+880246484 ms)
2025-02-18 00:26:08.665 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+880246485 ms)
2025-02-18 00:26:08.666 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47228)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+880246485 ms)
2025-02-18 00:26:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880264402 ms)
2025-02-18 00:26:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880264403 ms)
2025-02-18 00:26:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:26:25.400194+00:00', 'runtime': '880260.001', 'adc_vb_f': '12.744', 'adc_va_f': '1.065', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.741', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880264404 ms)
2025-02-18 00:26:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:25:25.400080+00:00', 'runtime': '880200.001', 'adc_vb_f': '12.75', 'adc_va_f': '1.068', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.744', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.735', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880264404 ms)
2025-02-18 00:27:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880324414 ms)
2025-02-18 00:27:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880324414 ms)
2025-02-18 00:27:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:27:25.400179+00:00', 'runtime': '880320.001', 'adc_vb_f': '12.747', 'adc_va_f': '1.059', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.741', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880324415 ms)
2025-02-18 00:27:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:26:25.400194+00:00', 'runtime': '880260.001', 'adc_vb_f': '12.744', 'adc_va_f': '1.065', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.741', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880324416 ms)
2025-02-18 00:28:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880384412 ms)
2025-02-18 00:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880384412 ms)
2025-02-18 00:28:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:28:25.402892+00:00', 'runtime': '880380.003', 'adc_vb_f': '12.741', 'adc_va_f': '1.056', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.741', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880384414 ms)
2025-02-18 00:28:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:27:25.400179+00:00', 'runtime': '880320.001', 'adc_vb_f': '12.747', 'adc_va_f': '1.059', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.741', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880384415 ms)
2025-02-18 00:29:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880444407 ms)
2025-02-18 00:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880444407 ms)
2025-02-18 00:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:29:25.400800+00:00', 'runtime': '880440.001', 'adc_vb_f': '12.738', 'adc_va_f': '1.05', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.738', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880444409 ms)
2025-02-18 00:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:28:25.402892+00:00', 'runtime': '880380.003', 'adc_vb_f': '12.741', 'adc_va_f': '1.056', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.741', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880444410 ms)
2025-02-18 00:30:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880504417 ms)
2025-02-18 00:30:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880504417 ms)
2025-02-18 00:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:30:25.400193+00:00', 'runtime': '880500.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.05', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.738', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880504419 ms)
2025-02-18 00:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:29:25.400800+00:00', 'runtime': '880440.001', 'adc_vb_f': '12.738', 'adc_va_f': '1.05', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.738', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880504420 ms)
2025-02-18 00:30:35.435 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880513254 ms)
2025-02-18 00:30:35.436 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880513255 ms)
2025-02-18 00:30:35.437 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:30:25.405064+00:00'} (T+880513257 ms)
2025-02-18 00:30:46.052 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+880523871 ms)
2025-02-18 00:30:46.052 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+880523871 ms)
2025-02-18 00:30:46.053 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47518)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+880523872 ms)
2025-02-18 00:31:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880564412 ms)
2025-02-18 00:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880564412 ms)
2025-02-18 00:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:31:25.400180+00:00', 'runtime': '880560.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.05', 'adc_vl_f': '12.735', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880564413 ms)
2025-02-18 00:31:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:30:25.400193+00:00', 'runtime': '880500.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.05', 'adc_vl_f': '12.738', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '9', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.738', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880564414 ms)
2025-02-18 00:31:35.434 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880573253 ms)
2025-02-18 00:31:35.435 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+880573254 ms)
2025-02-18 00:31:35.436 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:31:25.413127+00:00'} (T+880573255 ms)
2025-02-18 00:32:08.154 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+880605973 ms)
2025-02-18 00:32:08.154 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+880605973 ms)
2025-02-18 00:32:08.154 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+880605974 ms)
2025-02-18 00:32:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880624407 ms)
2025-02-18 00:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880624407 ms)
2025-02-18 00:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:32:25.400181+00:00', 'runtime': '880620.001', 'adc_vb_f': '12.732', 'adc_va_f': '1.053', 'adc_vl_f': '12.717', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.729', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880624411 ms)
2025-02-18 00:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:31:25.400180+00:00', 'runtime': '880560.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.05', 'adc_vl_f': '12.735', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.732', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880624411 ms)
2025-02-18 00:32:39.294 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+880637114 ms)
2025-02-18 00:32:39.295 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+880637114 ms)
2025-02-18 00:32:39.295 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47640)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+880637114 ms)
2025-02-18 00:33:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880684403 ms)
2025-02-18 00:33:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880684403 ms)
2025-02-18 00:33:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:33:25.400623+00:00', 'runtime': '880680.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.053', 'adc_vl_f': '12.729', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.72', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880684404 ms)
2025-02-18 00:33:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:32:25.400181+00:00', 'runtime': '880620.001', 'adc_vb_f': '12.732', 'adc_va_f': '1.053', 'adc_vl_f': '12.717', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.4', 'ahl_t': '1593.4', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.729', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880684405 ms)
2025-02-18 00:34:20.506 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+880738326 ms)
2025-02-18 00:34:20.507 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+880738326 ms)
2025-02-18 00:34:20.507 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47746)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+880738327 ms)
2025-02-18 00:34:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880744413 ms)
2025-02-18 00:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880744413 ms)
2025-02-18 00:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:34:25.400204+00:00', 'runtime': '880740.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.053', 'adc_vl_f': '12.735', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.72', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880744415 ms)
2025-02-18 00:34:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:33:25.400623+00:00', 'runtime': '880680.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.053', 'adc_vl_f': '12.729', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.72', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880744415 ms)
2025-02-18 00:35:25.578 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880803397 ms)
2025-02-18 00:35:25.578 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880803398 ms)
2025-02-18 00:35:25.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:35:25.400205+00:00', 'runtime': '880800.001', 'adc_vb_f': '12.732', 'adc_va_f': '1.05', 'adc_vl_f': '12.735', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.72', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880803402 ms)
2025-02-18 00:35:25.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:34:25.400204+00:00', 'runtime': '880740.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.053', 'adc_vl_f': '12.735', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.72', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880803402 ms)
2025-02-18 00:36:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880864403 ms)
2025-02-18 00:36:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880864404 ms)
2025-02-18 00:36:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:36:25.400220+00:00', 'runtime': '880860.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.05', 'adc_vl_f': '12.732', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.717', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880864406 ms)
2025-02-18 00:36:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:35:25.400205+00:00', 'runtime': '880800.001', 'adc_vb_f': '12.732', 'adc_va_f': '1.05', 'adc_vl_f': '12.735', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.72', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880864406 ms)
2025-02-18 00:37:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880924416 ms)
2025-02-18 00:37:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880924416 ms)
2025-02-18 00:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:37:25.400187+00:00', 'runtime': '880920.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.053', 'adc_vl_f': '12.732', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.717', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880924417 ms)
2025-02-18 00:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:36:25.400220+00:00', 'runtime': '880860.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.05', 'adc_vl_f': '12.732', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.717', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880924418 ms)
2025-02-18 00:37:37.864 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+880935683 ms)
2025-02-18 00:37:37.864 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+880935684 ms)
2025-02-18 00:37:37.865 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47962)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+880935684 ms)
2025-02-18 00:38:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+880984410 ms)
2025-02-18 00:38:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+880984411 ms)
2025-02-18 00:38:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:38:25.400159+00:00', 'runtime': '880980.001', 'adc_vb_f': '12.732', 'adc_va_f': '1.05', 'adc_vl_f': '12.704', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.717', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880984413 ms)
2025-02-18 00:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:37:25.400187+00:00', 'runtime': '880920.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.053', 'adc_vl_f': '12.732', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.717', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+880984413 ms)
2025-02-18 00:39:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881044402 ms)
2025-02-18 00:39:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881044402 ms)
2025-02-18 00:39:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:39:25.400176+00:00', 'runtime': '881040.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.056', 'adc_vl_f': '12.708', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.732', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881044404 ms)
2025-02-18 00:39:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:38:25.400159+00:00', 'runtime': '880980.001', 'adc_vb_f': '12.732', 'adc_va_f': '1.05', 'adc_vl_f': '12.704', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.735', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.717', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881044405 ms)
2025-02-18 00:39:31.101 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+881048920 ms)
2025-02-18 00:39:31.101 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+881048921 ms)
2025-02-18 00:39:31.102 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48082)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+881048921 ms)
2025-02-18 00:40:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881104412 ms)
2025-02-18 00:40:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881104413 ms)
2025-02-18 00:40:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:40:25.400215+00:00', 'runtime': '881100.001', 'adc_vb_f': '12.726', 'adc_va_f': '1.056', 'adc_vl_f': '12.701', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.729', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881104414 ms)
2025-02-18 00:40:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:39:25.400176+00:00', 'runtime': '881040.001', 'adc_vb_f': '12.735', 'adc_va_f': '1.056', 'adc_vl_f': '12.708', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.732', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881104414 ms)
2025-02-18 00:40:35.656 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881113475 ms)
2025-02-18 00:40:35.656 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881113476 ms)
2025-02-18 00:40:35.657 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:40:25.405061+00:00'} (T+881113477 ms)
2025-02-18 00:41:18.292 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+881156111 ms)
2025-02-18 00:41:18.292 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+881156112 ms)
2025-02-18 00:41:18.293 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48196)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+881156112 ms)
2025-02-18 00:41:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881164423 ms)
2025-02-18 00:41:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881164423 ms)
2025-02-18 00:41:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:41:25.402837+00:00', 'runtime': '881160.003', 'adc_vb_f': '12.726', 'adc_va_f': '1.059', 'adc_vl_f': '12.704', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.726', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881164424 ms)
2025-02-18 00:41:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:40:25.400215+00:00', 'runtime': '881100.001', 'adc_vb_f': '12.726', 'adc_va_f': '1.056', 'adc_vl_f': '12.701', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.729', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881164425 ms)
2025-02-18 00:41:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881173433 ms)
2025-02-18 00:41:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881173434 ms)
2025-02-18 00:41:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:41:25.400166+00:00'} (T+881173435 ms)
2025-02-18 00:42:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881224417 ms)
2025-02-18 00:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881224418 ms)
2025-02-18 00:42:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:42:25.400180+00:00', 'runtime': '881220.001', 'adc_vb_f': '12.723', 'adc_va_f': '1.056', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.726', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881224422 ms)
2025-02-18 00:42:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:41:25.402837+00:00', 'runtime': '881160.003', 'adc_vb_f': '12.726', 'adc_va_f': '1.059', 'adc_vl_f': '12.704', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.726', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881224422 ms)
2025-02-18 00:42:35.763 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881233583 ms)
2025-02-18 00:42:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881233583 ms)
2025-02-18 00:42:35.765 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:42:25.413122+00:00'} (T+881233584 ms)
2025-02-18 00:43:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881284410 ms)
2025-02-18 00:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881284411 ms)
2025-02-18 00:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:43:25.402650+00:00', 'runtime': '881280.003', 'adc_vb_f': '12.729', 'adc_va_f': '1.062', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.723', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881284415 ms)
2025-02-18 00:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:42:25.400180+00:00', 'runtime': '881220.001', 'adc_vb_f': '12.723', 'adc_va_f': '1.056', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.726', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881284415 ms)
2025-02-18 00:43:35.820 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881293640 ms)
2025-02-18 00:43:35.821 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881293640 ms)
2025-02-18 00:43:35.822 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:43:25.400023+00:00'} (T+881293641 ms)
2025-02-18 00:44:22.521 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+881340341 ms)
2025-02-18 00:44:22.522 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+881340341 ms)
2025-02-18 00:44:22.522 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+881340342 ms)
2025-02-18 00:44:22.523 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+881340343 ms)
2025-02-18 00:44:22.524 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+881340343 ms)
2025-02-18 00:44:22.524 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+881340343 ms)
2025-02-18 00:44:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881344420 ms)
2025-02-18 00:44:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881344421 ms)
2025-02-18 00:44:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:44:25.402620+00:00', 'runtime': '881340.003', 'adc_vb_f': '12.726', 'adc_va_f': '1.065', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.72', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881344424 ms)
2025-02-18 00:44:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:43:25.402650+00:00', 'runtime': '881280.003', 'adc_vb_f': '12.729', 'adc_va_f': '1.062', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.723', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881344425 ms)
2025-02-18 00:44:35.755 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881353574 ms)
2025-02-18 00:44:35.756 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881353575 ms)
2025-02-18 00:44:35.757 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:44:25.399995+00:00'} (T+881353576 ms)
2025-02-18 00:45:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881404412 ms)
2025-02-18 00:45:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881404412 ms)
2025-02-18 00:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:45:25.402656+00:00', 'runtime': '881400.003', 'adc_vb_f': '12.726', 'adc_va_f': '1.059', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.717', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881404413 ms)
2025-02-18 00:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:44:25.402620+00:00', 'runtime': '881340.003', 'adc_vb_f': '12.726', 'adc_va_f': '1.065', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.72', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881404416 ms)
2025-02-18 00:45:35.845 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881413665 ms)
2025-02-18 00:45:35.846 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881413665 ms)
2025-02-18 00:45:35.847 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:45:25.400033+00:00'} (T+881413666 ms)
2025-02-18 00:46:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881464421 ms)
2025-02-18 00:46:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881464421 ms)
2025-02-18 00:46:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:46:25.400187+00:00', 'runtime': '881460.001', 'adc_vb_f': '12.714', 'adc_va_f': '1.062', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881464422 ms)
2025-02-18 00:46:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:45:25.402656+00:00', 'runtime': '881400.003', 'adc_vb_f': '12.726', 'adc_va_f': '1.059', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.717', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881464423 ms)
2025-02-18 00:47:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881524401 ms)
2025-02-18 00:47:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881524401 ms)
2025-02-18 00:47:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:47:25.400172+00:00', 'runtime': '881520.001', 'adc_vb_f': '12.72', 'adc_va_f': '1.065', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881524402 ms)
2025-02-18 00:47:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:46:25.400187+00:00', 'runtime': '881460.001', 'adc_vb_f': '12.714', 'adc_va_f': '1.062', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.5', 'ahl_t': '1593.5', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881524403 ms)
2025-02-18 00:48:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881584413 ms)
2025-02-18 00:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881584413 ms)
2025-02-18 00:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:48:25.400187+00:00', 'runtime': '881580.001', 'adc_vb_f': '12.714', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881584414 ms)
2025-02-18 00:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:47:25.400172+00:00', 'runtime': '881520.001', 'adc_vb_f': '12.72', 'adc_va_f': '1.065', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881584414 ms)
2025-02-18 00:49:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881644405 ms)
2025-02-18 00:49:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881644406 ms)
2025-02-18 00:49:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:49:25.400266+00:00', 'runtime': '881640.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881644407 ms)
2025-02-18 00:49:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:48:25.400187+00:00', 'runtime': '881580.001', 'adc_vb_f': '12.714', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881644407 ms)
2025-02-18 00:49:35.641 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881653460 ms)
2025-02-18 00:49:35.641 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881653460 ms)
2025-02-18 00:49:35.642 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:49:25.400341+00:00'} (T+881653461 ms)
2025-02-18 00:50:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881704417 ms)
2025-02-18 00:50:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881704417 ms)
2025-02-18 00:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:50:25.400186+00:00', 'runtime': '881700.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881704419 ms)
2025-02-18 00:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:49:25.400266+00:00', 'runtime': '881640.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.714', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881704420 ms)
2025-02-18 00:50:35.705 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881713524 ms)
2025-02-18 00:50:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+881713525 ms)
2025-02-18 00:50:35.707 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:50:25.400159+00:00'} (T+881713526 ms)
2025-02-18 00:50:45.084 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+881722903 ms)
2025-02-18 00:50:45.084 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+881722903 ms)
2025-02-18 00:50:45.085 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48808)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+881722904 ms)
2025-02-18 00:51:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881764408 ms)
2025-02-18 00:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881764409 ms)
2025-02-18 00:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:51:25.400090+00:00', 'runtime': '881760.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881764410 ms)
2025-02-18 00:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:50:25.400186+00:00', 'runtime': '881700.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881764410 ms)
2025-02-18 00:52:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881824402 ms)
2025-02-18 00:52:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881824402 ms)
2025-02-18 00:52:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:52:25.400043+00:00', 'runtime': '881820.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881824403 ms)
2025-02-18 00:52:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:51:25.400090+00:00', 'runtime': '881760.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881824403 ms)
2025-02-18 00:53:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881884412 ms)
2025-02-18 00:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881884412 ms)
2025-02-18 00:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:53:25.400193+00:00', 'runtime': '881880.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881884413 ms)
2025-02-18 00:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:52:25.400043+00:00', 'runtime': '881820.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881884414 ms)
2025-02-18 00:54:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+881944421 ms)
2025-02-18 00:54:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+881944421 ms)
2025-02-18 00:54:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:54:25.402778+00:00', 'runtime': '881940.003', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881944422 ms)
2025-02-18 00:54:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:53:25.400193+00:00', 'runtime': '881880.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20131', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+881944423 ms)
2025-02-18 00:55:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882004415 ms)
2025-02-18 00:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882004415 ms)
2025-02-18 00:55:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:55:25.400939+00:00', 'runtime': '882000.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882004416 ms)
2025-02-18 00:55:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:54:25.402778+00:00', 'runtime': '881940.003', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882004417 ms)
2025-02-18 00:56:12.570 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+882050390 ms)
2025-02-18 00:56:12.571 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+882050390 ms)
2025-02-18 00:56:12.571 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49162)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882050391 ms)
2025-02-18 00:56:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882064405 ms)
2025-02-18 00:56:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882064405 ms)
2025-02-18 00:56:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:56:25.400169+00:00', 'runtime': '882060.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882064406 ms)
2025-02-18 00:56:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:55:25.400939+00:00', 'runtime': '882000.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.068', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882064407 ms)
2025-02-18 00:56:46.617 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+882084436 ms)
2025-02-18 00:56:46.617 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+882084436 ms)
2025-02-18 00:56:46.618 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882084437 ms)
2025-02-18 00:56:46.619 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+882084438 ms)
2025-02-18 00:56:46.619 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+882084438 ms)
2025-02-18 00:56:46.620 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882084439 ms)
2025-02-18 00:56:46.620 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+882084440 ms)
2025-02-18 00:56:46.621 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+882084440 ms)
2025-02-18 00:56:46.621 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882084440 ms)
2025-02-18 00:56:46.622 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+882084442 ms)
2025-02-18 00:56:46.623 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+882084442 ms)
2025-02-18 00:56:46.623 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882084442 ms)
2025-02-18 00:56:46.624 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+882084443 ms)
2025-02-18 00:56:46.624 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+882084444 ms)
2025-02-18 00:56:46.625 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882084444 ms)
2025-02-18 00:57:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882124415 ms)
2025-02-18 00:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882124415 ms)
2025-02-18 00:57:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:57:25.400176+00:00', 'runtime': '882120.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.379', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882124416 ms)
2025-02-18 00:57:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:56:25.400169+00:00', 'runtime': '882060.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882124417 ms)
2025-02-18 00:58:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882184408 ms)
2025-02-18 00:58:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882184409 ms)
2025-02-18 00:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:58:25.400169+00:00', 'runtime': '882180.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882184410 ms)
2025-02-18 00:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:57:25.400176+00:00', 'runtime': '882120.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.379', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882184410 ms)
2025-02-18 00:58:36.061 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882193880 ms)
2025-02-18 00:58:36.061 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882193880 ms)
2025-02-18 00:58:36.062 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:58:25.400346+00:00'} (T+882193881 ms)
2025-02-18 00:59:00.873 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+882218692 ms)
2025-02-18 00:59:00.873 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+882218693 ms)
2025-02-18 00:59:00.874 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49342)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882218693 ms)
2025-02-18 00:59:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882244417 ms)
2025-02-18 00:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882244418 ms)
2025-02-18 00:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 00:59:25.400264+00:00', 'runtime': '882240.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882244420 ms)
2025-02-18 00:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:58:25.400169+00:00', 'runtime': '882180.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882244421 ms)
2025-02-18 00:59:35.901 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882253720 ms)
2025-02-18 00:59:35.902 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882253721 ms)
2025-02-18 00:59:35.903 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 00:59:25.400271+00:00'} (T+882253722 ms)
2025-02-18 01:00:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882304411 ms)
2025-02-18 01:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882304412 ms)
2025-02-18 01:00:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:00:25.400135+00:00', 'runtime': '882300.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882304416 ms)
2025-02-18 01:00:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 00:59:25.400264+00:00', 'runtime': '882240.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882304417 ms)
2025-02-18 01:00:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882313433 ms)
2025-02-18 01:00:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882313433 ms)
2025-02-18 01:00:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:00:25.400415+00:00'} (T+882313434 ms)
2025-02-18 01:01:11.111 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+882348930 ms)
2025-02-18 01:01:11.111 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+882348931 ms)
2025-02-18 01:01:11.112 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49482)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882348931 ms)
2025-02-18 01:01:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882364404 ms)
2025-02-18 01:01:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882364405 ms)
2025-02-18 01:01:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:01:25.400186+00:00', 'runtime': '882360.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882364406 ms)
2025-02-18 01:01:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:00:25.400135+00:00', 'runtime': '882300.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.6', 'ahl_t': '1593.6', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882364406 ms)
2025-02-18 01:01:32.212 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+882370031 ms)
2025-02-18 01:01:32.212 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+882370031 ms)
2025-02-18 01:01:32.213 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49504)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882370032 ms)
2025-02-18 01:01:35.895 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882373714 ms)
2025-02-18 01:01:35.895 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882373714 ms)
2025-02-18 01:01:35.896 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:01:25.400202+00:00'} (T+882373715 ms)
2025-02-18 01:02:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882424416 ms)
2025-02-18 01:02:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882424417 ms)
2025-02-18 01:02:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:02:25.400197+00:00', 'runtime': '882420.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882424418 ms)
2025-02-18 01:02:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:01:25.400186+00:00', 'runtime': '882360.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882424418 ms)
2025-02-18 01:03:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882484410 ms)
2025-02-18 01:03:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882484410 ms)
2025-02-18 01:03:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:03:25.400486+00:00', 'runtime': '882480.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882484411 ms)
2025-02-18 01:03:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:02:25.400197+00:00', 'runtime': '882420.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882484411 ms)
2025-02-18 01:04:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882544403 ms)
2025-02-18 01:04:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882544403 ms)
2025-02-18 01:04:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:04:25.400257+00:00', 'runtime': '882540.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882544405 ms)
2025-02-18 01:04:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:03:25.400486+00:00', 'runtime': '882480.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882544405 ms)
2025-02-18 01:05:26.609 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882604428 ms)
2025-02-18 01:05:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882604429 ms)
2025-02-18 01:05:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:05:25.402750+00:00', 'runtime': '882600.003', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882604430 ms)
2025-02-18 01:05:26.613 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:04:25.400257+00:00', 'runtime': '882540.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882604432 ms)
2025-02-18 01:06:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882664404 ms)
2025-02-18 01:06:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882664404 ms)
2025-02-18 01:06:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:06:25.400847+00:00', 'runtime': '882660.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882664405 ms)
2025-02-18 01:06:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:05:25.402750+00:00', 'runtime': '882600.003', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882664406 ms)
2025-02-18 01:07:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882724413 ms)
2025-02-18 01:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882724414 ms)
2025-02-18 01:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:07:25.400279+00:00', 'runtime': '882720.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882724415 ms)
2025-02-18 01:07:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:06:25.400847+00:00', 'runtime': '882660.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882724415 ms)
2025-02-18 01:07:34.703 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+882732522 ms)
2025-02-18 01:07:34.703 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+882732522 ms)
2025-02-18 01:07:34.703 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49898)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882732523 ms)
2025-02-18 01:07:35.914 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882733733 ms)
2025-02-18 01:07:35.914 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882733733 ms)
2025-02-18 01:07:35.915 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:07:25.413055+00:00'} (T+882733734 ms)
2025-02-18 01:08:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882784407 ms)
2025-02-18 01:08:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882784407 ms)
2025-02-18 01:08:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:08:25.400194+00:00', 'runtime': '882780.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882784408 ms)
2025-02-18 01:08:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:07:25.400279+00:00', 'runtime': '882720.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882784411 ms)
2025-02-18 01:08:36.077 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882793896 ms)
2025-02-18 01:08:36.078 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882793897 ms)
2025-02-18 01:08:36.079 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:08:25.405140+00:00'} (T+882793898 ms)
2025-02-18 01:09:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882844417 ms)
2025-02-18 01:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882844418 ms)
2025-02-18 01:09:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:09:25.400166+00:00', 'runtime': '882840.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882844422 ms)
2025-02-18 01:09:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:08:25.400194+00:00', 'runtime': '882780.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882844422 ms)
2025-02-18 01:09:35.650 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882853469 ms)
2025-02-18 01:09:35.650 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882853470 ms)
2025-02-18 01:09:35.652 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:09:25.400212+00:00'} (T+882853471 ms)
2025-02-18 01:10:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882904410 ms)
2025-02-18 01:10:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882904411 ms)
2025-02-18 01:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:10:25.400158+00:00', 'runtime': '882900.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882904415 ms)
2025-02-18 01:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:09:25.400166+00:00', 'runtime': '882840.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882904415 ms)
2025-02-18 01:10:35.673 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882913493 ms)
2025-02-18 01:10:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+882913493 ms)
2025-02-18 01:10:35.675 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:10:25.400138+00:00'} (T+882913495 ms)
2025-02-18 01:11:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+882964420 ms)
2025-02-18 01:11:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+882964421 ms)
2025-02-18 01:11:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:11:25.400192+00:00', 'runtime': '882960.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882964425 ms)
2025-02-18 01:11:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:10:25.400158+00:00', 'runtime': '882900.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+882964425 ms)
2025-02-18 01:11:32.099 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+882969918 ms)
2025-02-18 01:11:32.099 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+882969918 ms)
2025-02-18 01:11:32.100 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50150)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882969919 ms)
2025-02-18 01:11:38.183 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+882976002 ms)
2025-02-18 01:11:38.183 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+882976003 ms)
2025-02-18 01:11:38.184 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50160)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+882976003 ms)
2025-02-18 01:12:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883024411 ms)
2025-02-18 01:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883024411 ms)
2025-02-18 01:12:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:12:25.400191+00:00', 'runtime': '883020.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883024413 ms)
2025-02-18 01:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:11:25.400192+00:00', 'runtime': '882960.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883024413 ms)
2025-02-18 01:12:38.333 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+883036152 ms)
2025-02-18 01:12:38.333 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+883036153 ms)
2025-02-18 01:12:38.334 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50224)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+883036153 ms)
2025-02-18 01:13:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883084407 ms)
2025-02-18 01:13:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883084407 ms)
2025-02-18 01:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:13:25.400187+00:00', 'runtime': '883080.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883084412 ms)
2025-02-18 01:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:12:25.400191+00:00', 'runtime': '883020.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883084413 ms)
2025-02-18 01:14:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883144416 ms)
2025-02-18 01:14:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883144417 ms)
2025-02-18 01:14:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:14:25.400255+00:00', 'runtime': '883140.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883144421 ms)
2025-02-18 01:14:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:13:25.400187+00:00', 'runtime': '883080.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.7', 'ahl_t': '1593.7', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883144421 ms)
2025-02-18 01:14:33.561 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+883151380 ms)
2025-02-18 01:14:33.561 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+883151381 ms)
2025-02-18 01:14:33.562 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50346)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+883151381 ms)
2025-02-18 01:15:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883204410 ms)
2025-02-18 01:15:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883204410 ms)
2025-02-18 01:15:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:15:25.400157+00:00', 'runtime': '883200.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.708', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883204414 ms)
2025-02-18 01:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:14:25.400255+00:00', 'runtime': '883140.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '7', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.711', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883204415 ms)
2025-02-18 01:16:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883264402 ms)
2025-02-18 01:16:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883264403 ms)
2025-02-18 01:16:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:16:25.400587+00:00', 'runtime': '883260.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.704', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883264404 ms)
2025-02-18 01:16:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:15:25.400157+00:00', 'runtime': '883200.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.708', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883264404 ms)
2025-02-18 01:17:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883324412 ms)
2025-02-18 01:17:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883324412 ms)
2025-02-18 01:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:17:25.400205+00:00', 'runtime': '883320.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.704', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883324413 ms)
2025-02-18 01:17:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:16:25.400587+00:00', 'runtime': '883260.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.704', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883324414 ms)
2025-02-18 01:17:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883333444 ms)
2025-02-18 01:17:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883333445 ms)
2025-02-18 01:17:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:17:25.413052+00:00'} (T+883333446 ms)
2025-02-18 01:18:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883384419 ms)
2025-02-18 01:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883384419 ms)
2025-02-18 01:18:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:18:25.402778+00:00', 'runtime': '883380.003', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.698', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883384420 ms)
2025-02-18 01:18:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:17:25.400205+00:00', 'runtime': '883320.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.704', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883384423 ms)
2025-02-18 01:18:35.823 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883393642 ms)
2025-02-18 01:18:35.823 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883393643 ms)
2025-02-18 01:18:35.824 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:18:25.400183+00:00'} (T+883393644 ms)
2025-02-18 01:19:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883444410 ms)
2025-02-18 01:19:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883444411 ms)
2025-02-18 01:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:19:25.400186+00:00', 'runtime': '883440.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.698', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883444412 ms)
2025-02-18 01:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:18:25.402778+00:00', 'runtime': '883380.003', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.698', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883444412 ms)
2025-02-18 01:19:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883453432 ms)
2025-02-18 01:19:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883453432 ms)
2025-02-18 01:19:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:19:25.400030+00:00'} (T+883453433 ms)
2025-02-18 01:19:35.932 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+883453751 ms)
2025-02-18 01:19:35.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+883453752 ms)
2025-02-18 01:19:35.933 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+883453753 ms)
2025-02-18 01:20:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883504404 ms)
2025-02-18 01:20:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883504404 ms)
2025-02-18 01:20:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:20:25.400167+00:00', 'runtime': '883500.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883504405 ms)
2025-02-18 01:20:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:19:25.400186+00:00', 'runtime': '883440.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.698', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883504406 ms)
2025-02-18 01:21:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883564413 ms)
2025-02-18 01:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883564414 ms)
2025-02-18 01:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:21:25.400270+00:00', 'runtime': '883560.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883564415 ms)
2025-02-18 01:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:20:25.400167+00:00', 'runtime': '883500.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883564415 ms)
2025-02-18 01:22:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883624407 ms)
2025-02-18 01:22:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883624407 ms)
2025-02-18 01:22:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:22:25.400192+00:00', 'runtime': '883620.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883624408 ms)
2025-02-18 01:22:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:21:25.400270+00:00', 'runtime': '883560.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883624409 ms)
2025-02-18 01:22:35.681 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883633501 ms)
2025-02-18 01:22:35.682 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883633501 ms)
2025-02-18 01:22:35.683 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:22:25.400353+00:00'} (T+883633502 ms)
2025-02-18 01:23:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883684417 ms)
2025-02-18 01:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883684418 ms)
2025-02-18 01:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:23:25.400195+00:00', 'runtime': '883680.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883684419 ms)
2025-02-18 01:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:22:25.400192+00:00', 'runtime': '883620.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883684419 ms)
2025-02-18 01:23:35.902 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883693721 ms)
2025-02-18 01:23:35.902 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883693722 ms)
2025-02-18 01:23:35.904 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:23:25.400181+00:00'} (T+883693723 ms)
2025-02-18 01:24:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883744411 ms)
2025-02-18 01:24:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883744412 ms)
2025-02-18 01:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:24:25.400179+00:00', 'runtime': '883740.001', 'adc_vb_f': '12.708', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883744414 ms)
2025-02-18 01:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:23:25.400195+00:00', 'runtime': '883680.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883744414 ms)
2025-02-18 01:24:35.489 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883753309 ms)
2025-02-18 01:24:35.490 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883753309 ms)
2025-02-18 01:24:35.491 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:24:25.400402+00:00'} (T+883753310 ms)
2025-02-18 01:25:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883804418 ms)
2025-02-18 01:25:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883804418 ms)
2025-02-18 01:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:25:25.400177+00:00', 'runtime': '883800.001', 'adc_vb_f': '12.708', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883804420 ms)
2025-02-18 01:25:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:24:25.400179+00:00', 'runtime': '883740.001', 'adc_vb_f': '12.708', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883804421 ms)
2025-02-18 01:25:33.424 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+883811243 ms)
2025-02-18 01:26:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883864412 ms)
2025-02-18 01:26:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883864412 ms)
2025-02-18 01:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:26:25.400246+00:00', 'runtime': '883860.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883864413 ms)
2025-02-18 01:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:25:25.400177+00:00', 'runtime': '883800.001', 'adc_vb_f': '12.708', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.708', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883864414 ms)
2025-02-18 01:27:08.571 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+883906391 ms)
2025-02-18 01:27:08.572 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+883906391 ms)
2025-02-18 01:27:08.572 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51162)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+883906391 ms)
2025-02-18 01:27:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883924404 ms)
2025-02-18 01:27:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883924404 ms)
2025-02-18 01:27:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:27:25.400149+00:00', 'runtime': '883920.001', 'adc_vb_f': '12.708', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883924405 ms)
2025-02-18 01:27:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:26:25.400246+00:00', 'runtime': '883860.001', 'adc_vb_f': '12.711', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883924406 ms)
2025-02-18 01:27:35.891 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883933711 ms)
2025-02-18 01:27:35.892 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883933711 ms)
2025-02-18 01:27:35.893 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:27:25.400328+00:00'} (T+883933712 ms)
2025-02-18 01:28:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+883984411 ms)
2025-02-18 01:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+883984411 ms)
2025-02-18 01:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:28:25.400530+00:00', 'runtime': '883980.001', 'adc_vb_f': '12.704', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883984412 ms)
2025-02-18 01:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:27:25.400149+00:00', 'runtime': '883920.001', 'adc_vb_f': '12.708', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.8', 'ahl_t': '1593.8', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+883984412 ms)
2025-02-18 01:28:35.776 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883993596 ms)
2025-02-18 01:28:35.777 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+883993596 ms)
2025-02-18 01:28:35.778 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:28:25.400426+00:00'} (T+883993597 ms)
2025-02-18 01:28:59.816 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884017635 ms)
2025-02-18 01:28:59.816 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884017635 ms)
2025-02-18 01:28:59.816 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51282)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884017636 ms)
2025-02-18 01:29:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884044418 ms)
2025-02-18 01:29:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884044418 ms)
2025-02-18 01:29:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:29:25.400196+00:00', 'runtime': '884040.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884044419 ms)
2025-02-18 01:29:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:28:25.400530+00:00', 'runtime': '883980.001', 'adc_vb_f': '12.704', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884044420 ms)
2025-02-18 01:29:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884053437 ms)
2025-02-18 01:29:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884053437 ms)
2025-02-18 01:29:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:29:25.400023+00:00'} (T+884053438 ms)
2025-02-18 01:29:48.931 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884066751 ms)
2025-02-18 01:29:48.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884066751 ms)
2025-02-18 01:29:48.932 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51334)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884066751 ms)
2025-02-18 01:29:50.995 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884068814 ms)
2025-02-18 01:29:50.995 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884068814 ms)
2025-02-18 01:29:50.995 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51340)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884068814 ms)
2025-02-18 01:30:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884104411 ms)
2025-02-18 01:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884104411 ms)
2025-02-18 01:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:30:25.400159+00:00', 'runtime': '884100.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.695', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884104412 ms)
2025-02-18 01:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:29:25.400196+00:00', 'runtime': '884040.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884104413 ms)
2025-02-18 01:30:36.070 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884113889 ms)
2025-02-18 01:30:36.071 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884113890 ms)
2025-02-18 01:30:36.071 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:30:25.400167+00:00'} (T+884113891 ms)
2025-02-18 01:31:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884164416 ms)
2025-02-18 01:31:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884164416 ms)
2025-02-18 01:31:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:31:25.400166+00:00', 'runtime': '884160.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.692', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884164417 ms)
2025-02-18 01:31:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:30:25.400159+00:00', 'runtime': '884100.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.695', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.695', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884164418 ms)
2025-02-18 01:31:35.786 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884173606 ms)
2025-02-18 01:31:35.787 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884173606 ms)
2025-02-18 01:31:35.788 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:31:25.400231+00:00'} (T+884173607 ms)
2025-02-18 01:31:39.133 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884176952 ms)
2025-02-18 01:31:39.133 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884176952 ms)
2025-02-18 01:31:39.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884176953 ms)
2025-02-18 01:32:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884224408 ms)
2025-02-18 01:32:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884224409 ms)
2025-02-18 01:32:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:32:25.400192+00:00', 'runtime': '884220.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.692', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.692', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884224410 ms)
2025-02-18 01:32:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:31:25.400166+00:00', 'runtime': '884160.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.692', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884224410 ms)
2025-02-18 01:32:35.745 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884233564 ms)
2025-02-18 01:32:35.745 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884233565 ms)
2025-02-18 01:32:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:32:25.400002+00:00'} (T+884233566 ms)
2025-02-18 01:33:01.311 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884259130 ms)
2025-02-18 01:33:01.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884259130 ms)
2025-02-18 01:33:01.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51544)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884259131 ms)
2025-02-18 01:33:16.417 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884274236 ms)
2025-02-18 01:33:16.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884274236 ms)
2025-02-18 01:33:16.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51560)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884274237 ms)
2025-02-18 01:33:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884284416 ms)
2025-02-18 01:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884284417 ms)
2025-02-18 01:33:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:33:25.402650+00:00', 'runtime': '884280.003', 'adc_vb_f': '12.701', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.692', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884284418 ms)
2025-02-18 01:33:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:32:25.400192+00:00', 'runtime': '884220.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.692', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.692', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884284419 ms)
2025-02-18 01:33:36.233 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884294052 ms)
2025-02-18 01:33:36.234 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884294053 ms)
2025-02-18 01:33:36.235 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:33:25.400033+00:00'} (T+884294054 ms)
2025-02-18 01:34:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884344403 ms)
2025-02-18 01:34:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884344403 ms)
2025-02-18 01:34:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:34:25.400186+00:00', 'runtime': '884340.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.695', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.698', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.689', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884344404 ms)
2025-02-18 01:34:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:33:25.402650+00:00', 'runtime': '884280.003', 'adc_vb_f': '12.701', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.701', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.692', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884344405 ms)
2025-02-18 01:34:28.577 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884346396 ms)
2025-02-18 01:34:28.577 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884346397 ms)
2025-02-18 01:34:28.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51638)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884346397 ms)
2025-02-18 01:34:36.014 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884353834 ms)
2025-02-18 01:34:36.015 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884353834 ms)
2025-02-18 01:34:36.021 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:34:25.400097+00:00'} (T+884353840 ms)
2025-02-18 01:35:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884404411 ms)
2025-02-18 01:35:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884404411 ms)
2025-02-18 01:35:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:35:25.401176+00:00', 'runtime': '884400.002', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.698', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.686', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884404413 ms)
2025-02-18 01:35:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:34:25.400186+00:00', 'runtime': '884340.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.695', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.698', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.689', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884404413 ms)
2025-02-18 01:35:36.120 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884413940 ms)
2025-02-18 01:35:36.121 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884413940 ms)
2025-02-18 01:35:36.122 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:35:25.400225+00:00'} (T+884413941 ms)
2025-02-18 01:35:36.667 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884414486 ms)
2025-02-18 01:35:36.667 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884414487 ms)
2025-02-18 01:35:36.668 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884414487 ms)
2025-02-18 01:35:36.668 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884414487 ms)
2025-02-18 01:35:36.668 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884414488 ms)
2025-02-18 01:35:36.669 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884414488 ms)
2025-02-18 01:35:36.669 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884414488 ms)
2025-02-18 01:35:36.669 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884414489 ms)
2025-02-18 01:35:36.670 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884414489 ms)
2025-02-18 01:35:36.670 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884414489 ms)
2025-02-18 01:35:36.670 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884414489 ms)
2025-02-18 01:35:36.670 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884414490 ms)
2025-02-18 01:35:36.671 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884414490 ms)
2025-02-18 01:35:36.671 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884414490 ms)
2025-02-18 01:35:36.671 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884414490 ms)
2025-02-18 01:35:36.671 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884414491 ms)
2025-02-18 01:35:36.672 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884414491 ms)
2025-02-18 01:35:36.672 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884414491 ms)
2025-02-18 01:35:36.674 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884414493 ms)
2025-02-18 01:35:36.674 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884414494 ms)
2025-02-18 01:35:36.675 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884414495 ms)
2025-02-18 01:36:21.816 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884459635 ms)
2025-02-18 01:36:21.816 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884459635 ms)
2025-02-18 01:36:21.816 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51770)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884459636 ms)
2025-02-18 01:36:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884464405 ms)
2025-02-18 01:36:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884464405 ms)
2025-02-18 01:36:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:36:25.400207+00:00', 'runtime': '884460.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.689', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.683', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884464406 ms)
2025-02-18 01:36:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:35:25.401176+00:00', 'runtime': '884400.002', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.698', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.698', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.686', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884464407 ms)
2025-02-18 01:37:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884524412 ms)
2025-02-18 01:37:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884524413 ms)
2025-02-18 01:37:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:37:25.400178+00:00', 'runtime': '884520.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.68', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.683', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884524414 ms)
2025-02-18 01:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:36:25.400207+00:00', 'runtime': '884460.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.689', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.683', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884524414 ms)
2025-02-18 01:38:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884584419 ms)
2025-02-18 01:38:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884584419 ms)
2025-02-18 01:38:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:38:25.400338+00:00', 'runtime': '884580.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.68', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.683', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884584420 ms)
2025-02-18 01:38:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:37:25.400178+00:00', 'runtime': '884520.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.68', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.683', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884584421 ms)
2025-02-18 01:38:35.717 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884593536 ms)
2025-02-18 01:38:35.717 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884593537 ms)
2025-02-18 01:38:35.724 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:38:25.400329+00:00'} (T+884593543 ms)
2025-02-18 01:38:50.053 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884607873 ms)
2025-02-18 01:38:50.054 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884607873 ms)
2025-02-18 01:38:50.054 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51928)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884607873 ms)
2025-02-18 01:39:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884644410 ms)
2025-02-18 01:39:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884644411 ms)
2025-02-18 01:39:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:39:25.400171+00:00', 'runtime': '884640.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.668', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.68', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884644411 ms)
2025-02-18 01:39:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:38:25.400338+00:00', 'runtime': '884580.001', 'adc_vb_f': '12.698', 'adc_va_f': '1.071', 'adc_vl_f': '12.68', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.683', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884644412 ms)
2025-02-18 01:39:35.903 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884653722 ms)
2025-02-18 01:39:35.903 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884653723 ms)
2025-02-18 01:39:35.910 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:39:25.400067+00:00'} (T+884653729 ms)
2025-02-18 01:40:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884704419 ms)
2025-02-18 01:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884704419 ms)
2025-02-18 01:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:40:25.400199+00:00', 'runtime': '884700.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.668', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.68', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884704420 ms)
2025-02-18 01:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:39:25.400171+00:00', 'runtime': '884640.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.668', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.68', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884704421 ms)
2025-02-18 01:40:35.810 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884713630 ms)
2025-02-18 01:40:35.811 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884713630 ms)
2025-02-18 01:40:35.812 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:40:25.400421+00:00'} (T+884713631 ms)
2025-02-18 01:41:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884764410 ms)
2025-02-18 01:41:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884764411 ms)
2025-02-18 01:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:41:25.400154+00:00', 'runtime': '884760.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.671', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.677', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884764412 ms)
2025-02-18 01:41:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:40:25.400199+00:00', 'runtime': '884700.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.668', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.695', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.68', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884764412 ms)
2025-02-18 01:41:36.003 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884773823 ms)
2025-02-18 01:41:36.004 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884773823 ms)
2025-02-18 01:41:36.009 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:41:25.399991+00:00'} (T+884773829 ms)
2025-02-18 01:41:44.273 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884782092 ms)
2025-02-18 01:41:44.273 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884782093 ms)
2025-02-18 01:41:44.274 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884782093 ms)
2025-02-18 01:41:44.275 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+884782094 ms)
2025-02-18 01:41:44.275 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+884782094 ms)
2025-02-18 01:41:44.275 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884782094 ms)
2025-02-18 01:42:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884824418 ms)
2025-02-18 01:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884824419 ms)
2025-02-18 01:42:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:42:25.400174+00:00', 'runtime': '884820.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.68', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.677', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884824423 ms)
2025-02-18 01:42:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:41:25.400154+00:00', 'runtime': '884760.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.671', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1593.9', 'ahl_t': '1593.9', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.677', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884824424 ms)
2025-02-18 01:42:35.742 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884833561 ms)
2025-02-18 01:42:35.743 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884833562 ms)
2025-02-18 01:42:35.747 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:42:25.400291+00:00'} (T+884833566 ms)
2025-02-18 01:43:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884884409 ms)
2025-02-18 01:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884884409 ms)
2025-02-18 01:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:43:25.400161+00:00', 'runtime': '884880.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.683', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.677', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884884413 ms)
2025-02-18 01:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:42:25.400174+00:00', 'runtime': '884820.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.68', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.677', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884884413 ms)
2025-02-18 01:43:28.448 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+884886268 ms)
2025-02-18 01:43:28.449 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+884886268 ms)
2025-02-18 01:43:28.449 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52226)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+884886269 ms)
2025-02-18 01:43:37.805 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884895625 ms)
2025-02-18 01:43:37.806 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884895625 ms)
2025-02-18 01:43:37.811 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:43:25.400043+00:00'} (T+884895630 ms)
2025-02-18 01:44:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+884944416 ms)
2025-02-18 01:44:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+884944416 ms)
2025-02-18 01:44:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:44:25.400207+00:00', 'runtime': '884940.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.677', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884944420 ms)
2025-02-18 01:44:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:43:25.400161+00:00', 'runtime': '884880.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.683', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.677', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+884944420 ms)
2025-02-18 01:44:35.812 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884953631 ms)
2025-02-18 01:44:35.813 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+884953632 ms)
2025-02-18 01:44:35.814 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:44:25.399999+00:00'} (T+884953634 ms)
2025-02-18 01:45:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885004409 ms)
2025-02-18 01:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885004410 ms)
2025-02-18 01:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:45:25.400184+00:00', 'runtime': '885000.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.686', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885004411 ms)
2025-02-18 01:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:44:25.400207+00:00', 'runtime': '884940.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.677', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885004413 ms)
2025-02-18 01:45:36.277 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885014096 ms)
2025-02-18 01:45:36.278 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885014097 ms)
2025-02-18 01:45:36.282 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:45:25.399993+00:00'} (T+885014101 ms)
2025-02-18 01:46:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885064416 ms)
2025-02-18 01:46:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885064416 ms)
2025-02-18 01:46:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:46:25.400201+00:00', 'runtime': '885060.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885064420 ms)
2025-02-18 01:46:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:45:25.400184+00:00', 'runtime': '885000.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.686', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.692', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885064421 ms)
2025-02-18 01:46:35.745 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885073565 ms)
2025-02-18 01:46:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885073565 ms)
2025-02-18 01:46:35.748 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:46:25.400020+00:00'} (T+885073567 ms)
2025-02-18 01:47:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885124405 ms)
2025-02-18 01:47:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885124405 ms)
2025-02-18 01:47:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:47:25.400173+00:00', 'runtime': '885120.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.665', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885124406 ms)
2025-02-18 01:47:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:46:25.400201+00:00', 'runtime': '885060.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885124407 ms)
2025-02-18 01:47:33.424 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
 (T+885131243 ms)
2025-02-18 01:48:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885184412 ms)
2025-02-18 01:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885184413 ms)
2025-02-18 01:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:48:25.400168+00:00', 'runtime': '885180.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.668', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885184417 ms)
2025-02-18 01:48:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:47:25.400173+00:00', 'runtime': '885120.001', 'adc_vb_f': '12.692', 'adc_va_f': '1.071', 'adc_vl_f': '12.665', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885184417 ms)
2025-02-18 01:49:23.977 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+885241796 ms)
2025-02-18 01:49:23.977 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+885241797 ms)
2025-02-18 01:49:23.978 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52612)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+885241797 ms)
2025-02-18 01:49:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885244418 ms)
2025-02-18 01:49:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885244419 ms)
2025-02-18 01:49:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:49:25.400534+00:00', 'runtime': '885240.001', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885244422 ms)
2025-02-18 01:49:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:48:25.400168+00:00', 'runtime': '885180.001', 'adc_vb_f': '12.695', 'adc_va_f': '1.071', 'adc_vl_f': '12.668', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885244423 ms)
2025-02-18 01:49:29.049 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+885246869 ms)
2025-02-18 01:49:29.050 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+885246869 ms)
2025-02-18 01:49:29.050 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52618)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+885246869 ms)
2025-02-18 01:49:35.492 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885253312 ms)
2025-02-18 01:49:35.493 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885253312 ms)
2025-02-18 01:49:35.498 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:49:25.400331+00:00'} (T+885253317 ms)
2025-02-18 01:50:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885304407 ms)
2025-02-18 01:50:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885304407 ms)
2025-02-18 01:50:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:50:25.400183+00:00', 'runtime': '885300.001', 'adc_vb_f': '12.683', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885304411 ms)
2025-02-18 01:50:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:49:25.400534+00:00', 'runtime': '885240.001', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885304412 ms)
2025-02-18 01:50:35.807 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885313627 ms)
2025-02-18 01:50:35.808 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885313627 ms)
2025-02-18 01:50:35.812 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:50:25.400324+00:00'} (T+885313632 ms)
2025-02-18 01:50:42.219 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+885320038 ms)
2025-02-18 01:50:42.219 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+885320038 ms)
2025-02-18 01:50:42.220 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52698)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+885320039 ms)
2025-02-18 01:51:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885364414 ms)
2025-02-18 01:51:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885364415 ms)
2025-02-18 01:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:51:25.400185+00:00', 'runtime': '885360.001', 'adc_vb_f': '12.683', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885364419 ms)
2025-02-18 01:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:50:25.400183+00:00', 'runtime': '885300.001', 'adc_vb_f': '12.683', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885364419 ms)
2025-02-18 01:51:36.011 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885373830 ms)
2025-02-18 01:51:36.012 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885373831 ms)
2025-02-18 01:51:36.016 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:51:25.399999+00:00'} (T+885373835 ms)
2025-02-18 01:52:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885424423 ms)
2025-02-18 01:52:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885424423 ms)
2025-02-18 01:52:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:52:25.400468+00:00', 'runtime': '885420.001', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.683', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885424427 ms)
2025-02-18 01:52:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:51:25.400185+00:00', 'runtime': '885360.001', 'adc_vb_f': '12.683', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.686', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885424428 ms)
2025-02-18 01:52:35.762 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885433582 ms)
2025-02-18 01:52:35.763 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885433582 ms)
2025-02-18 01:52:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:52:25.406183+00:00'} (T+885433583 ms)
2025-02-18 01:53:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885484412 ms)
2025-02-18 01:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885484413 ms)
2025-02-18 01:53:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:53:25.402622+00:00', 'runtime': '885480.003', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.683', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885484417 ms)
2025-02-18 01:53:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:52:25.400468+00:00', 'runtime': '885420.001', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.683', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885484417 ms)
2025-02-18 01:53:35.491 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+885493310 ms)
2025-02-18 01:53:35.491 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+885493311 ms)
2025-02-18 01:53:35.492 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52884)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+885493311 ms)
2025-02-18 01:54:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885544406 ms)
2025-02-18 01:54:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885544406 ms)
2025-02-18 01:54:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:54:25.400037+00:00', 'runtime': '885540.001', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.68', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885544409 ms)
2025-02-18 01:54:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:53:25.402622+00:00', 'runtime': '885480.003', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.683', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20132', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885544410 ms)
2025-02-18 01:55:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885604413 ms)
2025-02-18 01:55:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885604414 ms)
2025-02-18 01:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:55:25.400269+00:00', 'runtime': '885600.001', 'adc_vb_f': '12.68', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.68', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885604418 ms)
2025-02-18 01:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:54:25.400037+00:00', 'runtime': '885540.001', 'adc_vb_f': '12.686', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.68', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885604418 ms)
2025-02-18 01:55:42.749 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+885620569 ms)
2025-02-18 01:55:42.750 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+885620569 ms)
2025-02-18 01:55:42.750 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53020)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+885620569 ms)
2025-02-18 01:56:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885664404 ms)
2025-02-18 01:56:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885664404 ms)
2025-02-18 01:56:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:56:25.400201+00:00', 'runtime': '885660.001', 'adc_vb_f': '12.68', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885664408 ms)
2025-02-18 01:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:55:25.400269+00:00', 'runtime': '885600.001', 'adc_vb_f': '12.68', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.68', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.0', 'ahl_t': '1594.0', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885664408 ms)
2025-02-18 01:57:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885724409 ms)
2025-02-18 01:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885724410 ms)
2025-02-18 01:57:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:57:25.400595+00:00', 'runtime': '885720.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885724414 ms)
2025-02-18 01:57:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:56:25.400201+00:00', 'runtime': '885660.001', 'adc_vb_f': '12.68', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885724414 ms)
2025-02-18 01:58:25.026 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+885782845 ms)
2025-02-18 01:58:25.026 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+885782845 ms)
2025-02-18 01:58:25.027 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53198)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+885782846 ms)
2025-02-18 01:58:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885784417 ms)
2025-02-18 01:58:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885784418 ms)
2025-02-18 01:58:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:58:25.402783+00:00', 'runtime': '885780.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885784421 ms)
2025-02-18 01:58:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:57:25.400595+00:00', 'runtime': '885720.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885784421 ms)
2025-02-18 01:58:35.665 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885793484 ms)
2025-02-18 01:58:35.665 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885793485 ms)
2025-02-18 01:58:35.666 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 01:58:25.400144+00:00'} (T+885793486 ms)
2025-02-18 01:59:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885844406 ms)
2025-02-18 01:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885844407 ms)
2025-02-18 01:59:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 01:59:25.402819+00:00', 'runtime': '885840.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885844408 ms)
2025-02-18 01:59:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:58:25.402783+00:00', 'runtime': '885780.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885844408 ms)
2025-02-18 02:00:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885904414 ms)
2025-02-18 02:00:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885904414 ms)
2025-02-18 02:00:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:00:25.400189+00:00', 'runtime': '885900.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885904415 ms)
2025-02-18 02:00:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 01:59:25.402819+00:00', 'runtime': '885840.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885904416 ms)
2025-02-18 02:01:02.307 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+885940126 ms)
2025-02-18 02:01:02.307 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+885940126 ms)
2025-02-18 02:01:02.307 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53366)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+885940127 ms)
2025-02-18 02:01:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+885964421 ms)
2025-02-18 02:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+885964422 ms)
2025-02-18 02:01:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:01:25.400008+00:00', 'runtime': '885960.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885964423 ms)
2025-02-18 02:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:00:25.400189+00:00', 'runtime': '885900.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+885964423 ms)
2025-02-18 02:01:35.791 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885973610 ms)
2025-02-18 02:01:35.791 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+885973610 ms)
2025-02-18 02:01:35.792 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:01:25.409052+00:00'} (T+885973612 ms)
2025-02-18 02:02:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886024413 ms)
2025-02-18 02:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886024413 ms)
2025-02-18 02:02:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:02:25.402618+00:00', 'runtime': '886020.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886024415 ms)
2025-02-18 02:02:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:01:25.400008+00:00', 'runtime': '885960.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886024415 ms)
2025-02-18 02:02:35.749 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886033568 ms)
2025-02-18 02:02:35.749 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886033569 ms)
2025-02-18 02:02:35.750 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:02:25.400005+00:00'} (T+886033570 ms)
2025-02-18 02:03:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886084419 ms)
2025-02-18 02:03:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886084420 ms)
2025-02-18 02:03:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:03:25.402659+00:00', 'runtime': '886080.003', 'adc_vb_f': '12.677', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886084421 ms)
2025-02-18 02:03:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:02:25.402618+00:00', 'runtime': '886020.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886084422 ms)
2025-02-18 02:03:35.960 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886093780 ms)
2025-02-18 02:03:35.961 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886093780 ms)
2025-02-18 02:03:35.962 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:03:25.400033+00:00'} (T+886093781 ms)
2025-02-18 02:04:00.525 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+886118345 ms)
2025-02-18 02:04:00.526 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+886118345 ms)
2025-02-18 02:04:00.526 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886118346 ms)
2025-02-18 02:04:00.527 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+886118346 ms)
2025-02-18 02:04:00.527 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+886118347 ms)
2025-02-18 02:04:00.528 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886118347 ms)
2025-02-18 02:04:00.529 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+886118348 ms)
2025-02-18 02:04:00.529 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+886118348 ms)
2025-02-18 02:04:00.530 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886118349 ms)
2025-02-18 02:04:00.530 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+886118350 ms)
2025-02-18 02:04:00.530 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+886118350 ms)
2025-02-18 02:04:00.531 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886118350 ms)
2025-02-18 02:04:00.534 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+886118353 ms)
2025-02-18 02:04:00.534 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+886118353 ms)
2025-02-18 02:04:00.535 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886118354 ms)
2025-02-18 02:04:00.536 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+886118355 ms)
2025-02-18 02:04:00.536 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+886118355 ms)
2025-02-18 02:04:00.536 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886118356 ms)
2025-02-18 02:04:26.609 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886144428 ms)
2025-02-18 02:04:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886144429 ms)
2025-02-18 02:04:26.612 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:04:25.403175+00:00', 'runtime': '886140.004', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886144431 ms)
2025-02-18 02:04:26.612 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:03:25.402659+00:00', 'runtime': '886080.003', 'adc_vb_f': '12.677', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.677', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886144431 ms)
2025-02-18 02:05:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886204407 ms)
2025-02-18 02:05:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886204407 ms)
2025-02-18 02:05:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:05:25.400170+00:00', 'runtime': '886200.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886204408 ms)
2025-02-18 02:05:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:04:25.403175+00:00', 'runtime': '886140.004', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886204409 ms)
2025-02-18 02:06:02.763 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886240582 ms)
2025-02-18 02:06:02.763 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886240582 ms)
2025-02-18 02:06:02.764 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53696)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886240583 ms)
2025-02-18 02:06:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886264413 ms)
2025-02-18 02:06:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886264413 ms)
2025-02-18 02:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:06:25.400208+00:00', 'runtime': '886260.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886264414 ms)
2025-02-18 02:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:05:25.400170+00:00', 'runtime': '886200.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886264415 ms)
2025-02-18 02:06:57.922 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886295741 ms)
2025-02-18 02:06:57.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886295741 ms)
2025-02-18 02:06:57.922 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53754)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886295742 ms)
2025-02-18 02:07:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886324407 ms)
2025-02-18 02:07:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886324408 ms)
2025-02-18 02:07:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:07:25.400220+00:00', 'runtime': '886320.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886324409 ms)
2025-02-18 02:07:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:06:25.400208+00:00', 'runtime': '886260.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886324409 ms)
2025-02-18 02:07:35.419 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886333238 ms)
2025-02-18 02:07:35.420 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886333239 ms)
2025-02-18 02:07:35.421 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:07:25.400337+00:00'} (T+886333240 ms)
2025-02-18 02:07:46.047 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886343866 ms)
2025-02-18 02:07:46.047 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886343866 ms)
2025-02-18 02:07:46.048 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53806)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886343867 ms)
2025-02-18 02:08:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886384414 ms)
2025-02-18 02:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886384414 ms)
2025-02-18 02:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:08:25.400176+00:00', 'runtime': '886380.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886384415 ms)
2025-02-18 02:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:07:25.400220+00:00', 'runtime': '886320.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886384416 ms)
2025-02-18 02:09:15.236 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886433056 ms)
2025-02-18 02:09:15.237 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886433056 ms)
2025-02-18 02:09:15.237 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53902)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886433057 ms)
2025-02-18 02:09:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886444409 ms)
2025-02-18 02:09:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886444409 ms)
2025-02-18 02:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:09:25.400176+00:00', 'runtime': '886440.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886444410 ms)
2025-02-18 02:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:08:25.400176+00:00', 'runtime': '886380.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886444411 ms)
2025-02-18 02:09:49.327 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886467146 ms)
2025-02-18 02:09:49.327 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886467147 ms)
2025-02-18 02:09:49.328 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53940)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886467147 ms)
2025-02-18 02:10:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886504414 ms)
2025-02-18 02:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886504415 ms)
2025-02-18 02:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:10:25.400179+00:00', 'runtime': '886500.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886504416 ms)
2025-02-18 02:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:09:25.400176+00:00', 'runtime': '886440.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.1', 'ahl_t': '1594.1', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886504416 ms)
2025-02-18 02:11:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886564409 ms)
2025-02-18 02:11:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886564409 ms)
2025-02-18 02:11:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:11:25.400204+00:00', 'runtime': '886560.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886564411 ms)
2025-02-18 02:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:10:25.400179+00:00', 'runtime': '886500.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886564411 ms)
2025-02-18 02:12:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886624412 ms)
2025-02-18 02:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886624413 ms)
2025-02-18 02:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:12:25.400215+00:00', 'runtime': '886620.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886624414 ms)
2025-02-18 02:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:11:25.400204+00:00', 'runtime': '886560.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886624414 ms)
2025-02-18 02:12:46.622 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886644442 ms)
2025-02-18 02:12:46.623 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886644442 ms)
2025-02-18 02:12:46.623 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54124)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886644443 ms)
2025-02-18 02:13:02.725 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886660545 ms)
2025-02-18 02:13:02.726 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886660545 ms)
2025-02-18 02:13:02.726 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54144)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886660545 ms)
2025-02-18 02:13:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886684403 ms)
2025-02-18 02:13:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886684403 ms)
2025-02-18 02:13:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:13:25.400162+00:00', 'runtime': '886680.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886684404 ms)
2025-02-18 02:13:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:12:25.400215+00:00', 'runtime': '886620.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886684405 ms)
2025-02-18 02:14:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886744409 ms)
2025-02-18 02:14:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886744409 ms)
2025-02-18 02:14:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:14:25.400269+00:00', 'runtime': '886740.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886744410 ms)
2025-02-18 02:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:13:25.400162+00:00', 'runtime': '886680.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886744411 ms)
2025-02-18 02:15:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886804416 ms)
2025-02-18 02:15:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886804417 ms)
2025-02-18 02:15:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:15:25.400198+00:00', 'runtime': '886800.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886804418 ms)
2025-02-18 02:15:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:14:25.400269+00:00', 'runtime': '886740.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886804419 ms)
2025-02-18 02:16:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886864408 ms)
2025-02-18 02:16:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886864409 ms)
2025-02-18 02:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:16:25.400185+00:00', 'runtime': '886860.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886864410 ms)
2025-02-18 02:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:15:25.400198+00:00', 'runtime': '886800.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886864411 ms)
2025-02-18 02:16:52.071 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886889890 ms)
2025-02-18 02:16:52.071 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886889890 ms)
2025-02-18 02:16:52.072 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54394)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886889891 ms)
2025-02-18 02:17:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886924416 ms)
2025-02-18 02:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886924417 ms)
2025-02-18 02:17:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:17:25.400204+00:00', 'runtime': '886920.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886924418 ms)
2025-02-18 02:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:16:25.400185+00:00', 'runtime': '886860.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886924418 ms)
2025-02-18 02:17:35.621 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886933441 ms)
2025-02-18 02:17:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+886933441 ms)
2025-02-18 02:17:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:17:25.400339+00:00'} (T+886933442 ms)
2025-02-18 02:17:37.213 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886935032 ms)
2025-02-18 02:17:37.213 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886935033 ms)
2025-02-18 02:17:37.214 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54442)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886935033 ms)
2025-02-18 02:18:09.322 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+886967142 ms)
2025-02-18 02:18:09.323 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+886967142 ms)
2025-02-18 02:18:09.323 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54478)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+886967142 ms)
2025-02-18 02:18:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+886984408 ms)
2025-02-18 02:18:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+886984409 ms)
2025-02-18 02:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:18:25.400184+00:00', 'runtime': '886980.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886984410 ms)
2025-02-18 02:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:17:25.400204+00:00', 'runtime': '886920.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+886984410 ms)
2025-02-18 02:19:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887044416 ms)
2025-02-18 02:19:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887044417 ms)
2025-02-18 02:19:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:19:25.400171+00:00', 'runtime': '887040.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887044418 ms)
2025-02-18 02:19:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:18:25.400184+00:00', 'runtime': '886980.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887044418 ms)
2025-02-18 02:20:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887104406 ms)
2025-02-18 02:20:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887104406 ms)
2025-02-18 02:20:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:20:25.400173+00:00', 'runtime': '887100.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887104407 ms)
2025-02-18 02:20:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:19:25.400171+00:00', 'runtime': '887040.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887104408 ms)
2025-02-18 02:20:35.486 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887113306 ms)
2025-02-18 02:20:35.487 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887113306 ms)
2025-02-18 02:20:35.488 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:20:25.400344+00:00'} (T+887113307 ms)
2025-02-18 02:21:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887164412 ms)
2025-02-18 02:21:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887164412 ms)
2025-02-18 02:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:21:25.400184+00:00', 'runtime': '887160.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887164413 ms)
2025-02-18 02:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:20:25.400173+00:00', 'runtime': '887100.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887164414 ms)
2025-02-18 02:22:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887224417 ms)
2025-02-18 02:22:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887224418 ms)
2025-02-18 02:22:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:22:25.400201+00:00', 'runtime': '887220.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887224420 ms)
2025-02-18 02:22:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:21:25.400184+00:00', 'runtime': '887160.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887224420 ms)
2025-02-18 02:23:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887284425 ms)
2025-02-18 02:23:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887284425 ms)
2025-02-18 02:23:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:23:25.400162+00:00', 'runtime': '887280.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887284426 ms)
2025-02-18 02:23:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:22:25.400201+00:00', 'runtime': '887220.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887284426 ms)
2025-02-18 02:23:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887293436 ms)
2025-02-18 02:23:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887293436 ms)
2025-02-18 02:23:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:23:25.405056+00:00'} (T+887293437 ms)
2025-02-18 02:24:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887344414 ms)
2025-02-18 02:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887344414 ms)
2025-02-18 02:24:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:24:25.400150+00:00', 'runtime': '887340.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887344415 ms)
2025-02-18 02:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:23:25.400162+00:00', 'runtime': '887280.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.2', 'ahl_t': '1594.2', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '0.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887344416 ms)
2025-02-18 02:24:35.447 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887353266 ms)
2025-02-18 02:24:35.448 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887353267 ms)
2025-02-18 02:24:35.449 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:24:25.413121+00:00'} (T+887353268 ms)
2025-02-18 02:25:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887404419 ms)
2025-02-18 02:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887404420 ms)
2025-02-18 02:25:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:25:25.402858+00:00', 'runtime': '887400.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887404421 ms)
2025-02-18 02:25:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:24:25.400150+00:00', 'runtime': '887340.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887404421 ms)
2025-02-18 02:25:51.953 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+887429772 ms)
2025-02-18 02:25:51.953 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+887429772 ms)
2025-02-18 02:25:51.954 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54980)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+887429773 ms)
2025-02-18 02:26:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887464409 ms)
2025-02-18 02:26:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887464409 ms)
2025-02-18 02:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:26:25.400505+00:00', 'runtime': '887460.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887464410 ms)
2025-02-18 02:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:25:25.402858+00:00', 'runtime': '887400.003', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887464410 ms)
2025-02-18 02:27:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887524416 ms)
2025-02-18 02:27:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887524417 ms)
2025-02-18 02:27:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:27:25.400187+00:00', 'runtime': '887520.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887524418 ms)
2025-02-18 02:27:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:26:25.400505+00:00', 'runtime': '887460.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887524418 ms)
2025-02-18 02:27:36.474 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887534293 ms)
2025-02-18 02:27:36.474 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887534293 ms)
2025-02-18 02:27:36.476 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:27:25.400338+00:00'} (T+887534295 ms)
2025-02-18 02:28:00.186 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+887558006 ms)
2025-02-18 02:28:00.187 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+887558006 ms)
2025-02-18 02:28:00.187 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55116)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+887558007 ms)
2025-02-18 02:28:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887584407 ms)
2025-02-18 02:28:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887584407 ms)
2025-02-18 02:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:28:25.400170+00:00', 'runtime': '887580.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887584408 ms)
2025-02-18 02:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:27:25.400187+00:00', 'runtime': '887520.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887584408 ms)
2025-02-18 02:28:59.338 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+887617157 ms)
2025-02-18 02:28:59.338 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+887617157 ms)
2025-02-18 02:28:59.339 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55180)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+887617158 ms)
2025-02-18 02:29:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887644415 ms)
2025-02-18 02:29:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887644415 ms)
2025-02-18 02:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:29:25.400197+00:00', 'runtime': '887640.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887644416 ms)
2025-02-18 02:29:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:28:25.400170+00:00', 'runtime': '887580.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.674', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887644417 ms)
2025-02-18 02:30:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887704402 ms)
2025-02-18 02:30:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887704402 ms)
2025-02-18 02:30:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:30:25.400158+00:00', 'runtime': '887700.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887704403 ms)
2025-02-18 02:30:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:29:25.400197+00:00', 'runtime': '887640.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887704404 ms)
2025-02-18 02:31:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887764410 ms)
2025-02-18 02:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887764411 ms)
2025-02-18 02:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:31:25.400211+00:00', 'runtime': '887760.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887764412 ms)
2025-02-18 02:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:30:25.400158+00:00', 'runtime': '887700.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887764412 ms)
2025-02-18 02:31:33.423 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+887771243 ms)
2025-02-18 02:32:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887824415 ms)
2025-02-18 02:32:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887824416 ms)
2025-02-18 02:32:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:32:25.400209+00:00', 'runtime': '887820.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887824417 ms)
2025-02-18 02:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:31:25.400211+00:00', 'runtime': '887760.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887824417 ms)
2025-02-18 02:33:00.738 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+887858557 ms)
2025-02-18 02:33:00.738 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+887858558 ms)
2025-02-18 02:33:00.739 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55430)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+887858558 ms)
2025-02-18 02:33:03.824 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+887861644 ms)
2025-02-18 02:33:03.825 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+887861644 ms)
2025-02-18 02:33:03.825 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55434)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+887861644 ms)
2025-02-18 02:33:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887884407 ms)
2025-02-18 02:33:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887884407 ms)
2025-02-18 02:33:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:33:25.400177+00:00', 'runtime': '887880.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887884408 ms)
2025-02-18 02:33:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:32:25.400209+00:00', 'runtime': '887820.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887884409 ms)
2025-02-18 02:34:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+887944413 ms)
2025-02-18 02:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+887944413 ms)
2025-02-18 02:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:34:25.400261+00:00', 'runtime': '887940.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887944414 ms)
2025-02-18 02:34:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:33:25.400177+00:00', 'runtime': '887880.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+887944415 ms)
2025-02-18 02:34:35.637 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887953456 ms)
2025-02-18 02:34:35.638 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+887953457 ms)
2025-02-18 02:34:35.639 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:34:25.400348+00:00'} (T+887953458 ms)
2025-02-18 02:35:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888004420 ms)
2025-02-18 02:35:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888004421 ms)
2025-02-18 02:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:35:25.400217+00:00', 'runtime': '888000.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.668', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888004422 ms)
2025-02-18 02:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:34:25.400261+00:00', 'runtime': '887940.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.671', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888004422 ms)
2025-02-18 02:35:35.796 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888013616 ms)
2025-02-18 02:35:35.798 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888013617 ms)
2025-02-18 02:35:35.807 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:35:25.400217+00:00'} (T+888013626 ms)
2025-02-18 02:36:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888064409 ms)
2025-02-18 02:36:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888064410 ms)
2025-02-18 02:36:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:36:25.400191+00:00', 'runtime': '888060.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888064411 ms)
2025-02-18 02:36:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:35:25.400217+00:00', 'runtime': '888000.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.668', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888064411 ms)
2025-02-18 02:37:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888124413 ms)
2025-02-18 02:37:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888124413 ms)
2025-02-18 02:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:37:25.400188+00:00', 'runtime': '888120.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888124414 ms)
2025-02-18 02:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:36:25.400191+00:00', 'runtime': '888060.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888124415 ms)
2025-02-18 02:38:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888184417 ms)
2025-02-18 02:38:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888184417 ms)
2025-02-18 02:38:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:38:25.400180+00:00', 'runtime': '888180.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888184418 ms)
2025-02-18 02:38:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:37:25.400188+00:00', 'runtime': '888120.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.3', 'ahl_t': '1594.3', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888184419 ms)
2025-02-18 02:38:35.730 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888193549 ms)
2025-02-18 02:38:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888193549 ms)
2025-02-18 02:38:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:38:25.400302+00:00'} (T+888193551 ms)
2025-02-18 02:39:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888244404 ms)
2025-02-18 02:39:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888244405 ms)
2025-02-18 02:39:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:39:25.400155+00:00', 'runtime': '888240.001', 'adc_vb_f': '12.671', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888244409 ms)
2025-02-18 02:39:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:38:25.400180+00:00', 'runtime': '888180.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888244409 ms)
2025-02-18 02:39:36.455 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888254275 ms)
2025-02-18 02:39:36.456 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888254275 ms)
2025-02-18 02:39:36.458 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:39:25.400062+00:00'} (T+888254277 ms)
2025-02-18 02:40:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888304410 ms)
2025-02-18 02:40:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888304411 ms)
2025-02-18 02:40:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:40:25.400176+00:00', 'runtime': '888300.001', 'adc_vb_f': '12.671', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888304412 ms)
2025-02-18 02:40:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:39:25.400155+00:00', 'runtime': '888240.001', 'adc_vb_f': '12.671', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888304412 ms)
2025-02-18 02:40:36.132 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888313951 ms)
2025-02-18 02:40:36.132 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888313952 ms)
2025-02-18 02:40:36.134 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:40:25.400380+00:00'} (T+888313953 ms)
2025-02-18 02:41:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888364417 ms)
2025-02-18 02:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888364418 ms)
2025-02-18 02:41:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:41:25.400164+00:00', 'runtime': '888360.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888364418 ms)
2025-02-18 02:41:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:40:25.400176+00:00', 'runtime': '888300.001', 'adc_vb_f': '12.671', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888364419 ms)
2025-02-18 02:41:36.526 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888374346 ms)
2025-02-18 02:41:36.527 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888374346 ms)
2025-02-18 02:41:36.529 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:41:25.400158+00:00'} (T+888374348 ms)
2025-02-18 02:42:15.593 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+888413412 ms)
2025-02-18 02:42:15.593 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+888413413 ms)
2025-02-18 02:42:15.594 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56026)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888413413 ms)
2025-02-18 02:42:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888424406 ms)
2025-02-18 02:42:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888424406 ms)
2025-02-18 02:42:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:42:25.400185+00:00', 'runtime': '888420.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888424407 ms)
2025-02-18 02:42:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:41:25.400164+00:00', 'runtime': '888360.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888424408 ms)
2025-02-18 02:43:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888484410 ms)
2025-02-18 02:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888484410 ms)
2025-02-18 02:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:43:25.400189+00:00', 'runtime': '888480.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888484411 ms)
2025-02-18 02:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:42:25.400185+00:00', 'runtime': '888420.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888484411 ms)
2025-02-18 02:44:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888544412 ms)
2025-02-18 02:44:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888544412 ms)
2025-02-18 02:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:44:25.400168+00:00', 'runtime': '888540.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.671', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888544413 ms)
2025-02-18 02:44:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:43:25.400189+00:00', 'runtime': '888480.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888544414 ms)
2025-02-18 02:44:36.056 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888553875 ms)
2025-02-18 02:44:36.057 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888553876 ms)
2025-02-18 02:44:36.059 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:44:25.400322+00:00'} (T+888553878 ms)
2025-02-18 02:45:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888604418 ms)
2025-02-18 02:45:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888604419 ms)
2025-02-18 02:45:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:45:25.400151+00:00', 'runtime': '888600.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.671', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888604420 ms)
2025-02-18 02:45:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:44:25.400168+00:00', 'runtime': '888540.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.671', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888604420 ms)
2025-02-18 02:45:35.723 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888613543 ms)
2025-02-18 02:45:35.724 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888613543 ms)
2025-02-18 02:45:35.725 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:45:25.400231+00:00'} (T+888613544 ms)
2025-02-18 02:46:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888664405 ms)
2025-02-18 02:46:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888664405 ms)
2025-02-18 02:46:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:46:25.400189+00:00', 'runtime': '888660.001', 'adc_vb_f': '12.668', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888664406 ms)
2025-02-18 02:46:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:45:25.400151+00:00', 'runtime': '888600.001', 'adc_vb_f': '12.674', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.671', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888664407 ms)
2025-02-18 02:46:35.652 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888673471 ms)
2025-02-18 02:46:35.652 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888673471 ms)
2025-02-18 02:46:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:46:25.400129+00:00'} (T+888673479 ms)
2025-02-18 02:46:36.992 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+888674812 ms)
2025-02-18 02:46:36.993 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+888674812 ms)
2025-02-18 02:46:36.993 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56314)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888674813 ms)
2025-02-18 02:47:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888724410 ms)
2025-02-18 02:47:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888724410 ms)
2025-02-18 02:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:47:25.400192+00:00', 'runtime': '888720.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888724411 ms)
2025-02-18 02:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:46:25.400189+00:00', 'runtime': '888660.001', 'adc_vb_f': '12.668', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888724411 ms)
2025-02-18 02:47:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888733433 ms)
2025-02-18 02:47:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888733433 ms)
2025-02-18 02:47:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:47:25.400129+00:00'} (T+888733437 ms)
2025-02-18 02:47:43.073 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740892 ms)
2025-02-18 02:47:43.073 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740893 ms)
2025-02-18 02:47:43.074 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740893 ms)
2025-02-18 02:47:43.075 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740894 ms)
2025-02-18 02:47:43.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740895 ms)
2025-02-18 02:47:43.076 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740895 ms)
2025-02-18 02:47:43.077 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740896 ms)
2025-02-18 02:47:43.077 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740896 ms)
2025-02-18 02:47:43.078 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740897 ms)
2025-02-18 02:47:43.079 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740898 ms)
2025-02-18 02:47:43.079 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740898 ms)
2025-02-18 02:47:43.079 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740899 ms)
2025-02-18 02:47:43.080 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740900 ms)
2025-02-18 02:47:43.080 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740900 ms)
2025-02-18 02:47:43.081 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740900 ms)
2025-02-18 02:47:43.082 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740901 ms)
2025-02-18 02:47:43.082 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740902 ms)
2025-02-18 02:47:43.083 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740902 ms)
2025-02-18 02:47:43.084 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740903 ms)
2025-02-18 02:47:43.084 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740903 ms)
2025-02-18 02:47:43.085 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740904 ms)
2025-02-18 02:47:43.086 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740905 ms)
2025-02-18 02:47:43.086 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740905 ms)
2025-02-18 02:47:43.087 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740906 ms)
2025-02-18 02:47:43.088 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888740907 ms)
2025-02-18 02:47:43.088 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888740907 ms)
2025-02-18 02:47:43.089 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888740908 ms)
2025-02-18 02:48:21.134 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778953 ms)
2025-02-18 02:48:21.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778953 ms)
2025-02-18 02:48:21.135 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778954 ms)
2025-02-18 02:48:21.136 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778955 ms)
2025-02-18 02:48:21.136 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778955 ms)
2025-02-18 02:48:21.136 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778956 ms)
2025-02-18 02:48:21.137 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778957 ms)
2025-02-18 02:48:21.138 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778957 ms)
2025-02-18 02:48:21.138 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778957 ms)
2025-02-18 02:48:21.139 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778958 ms)
2025-02-18 02:48:21.139 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778959 ms)
2025-02-18 02:48:21.140 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778959 ms)
2025-02-18 02:48:21.141 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778960 ms)
2025-02-18 02:48:21.141 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778960 ms)
2025-02-18 02:48:21.142 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778961 ms)
2025-02-18 02:48:21.143 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778962 ms)
2025-02-18 02:48:21.143 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778962 ms)
2025-02-18 02:48:21.143 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778963 ms)
2025-02-18 02:48:21.144 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778963 ms)
2025-02-18 02:48:21.144 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778964 ms)
2025-02-18 02:48:21.145 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778964 ms)
2025-02-18 02:48:21.146 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778965 ms)
2025-02-18 02:48:21.146 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778965 ms)
2025-02-18 02:48:21.147 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778966 ms)
2025-02-18 02:48:21.148 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+888778967 ms)
2025-02-18 02:48:21.148 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+888778967 ms)
2025-02-18 02:48:21.148 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+888778968 ms)
2025-02-18 02:48:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888784415 ms)
2025-02-18 02:48:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888784415 ms)
2025-02-18 02:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:48:25.400202+00:00', 'runtime': '888780.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888784416 ms)
2025-02-18 02:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:47:25.400192+00:00', 'runtime': '888720.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888784417 ms)
2025-02-18 02:48:35.663 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888793482 ms)
2025-02-18 02:48:35.664 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888793483 ms)
2025-02-18 02:48:35.668 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:48:25.400197+00:00'} (T+888793487 ms)
2025-02-18 02:49:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888844404 ms)
2025-02-18 02:49:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888844405 ms)
2025-02-18 02:49:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:49:25.400197+00:00', 'runtime': '888840.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888844406 ms)
2025-02-18 02:49:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:48:25.400202+00:00', 'runtime': '888780.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.659', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888844406 ms)
2025-02-18 02:49:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888853435 ms)
2025-02-18 02:49:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888853436 ms)
2025-02-18 02:49:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:49:25.400151+00:00'} (T+888853442 ms)
2025-02-18 02:50:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888904410 ms)
2025-02-18 02:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888904411 ms)
2025-02-18 02:50:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:50:25.400207+00:00', 'runtime': '888900.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888904412 ms)
2025-02-18 02:50:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:49:25.400197+00:00', 'runtime': '888840.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888904412 ms)
2025-02-18 02:50:35.779 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888913599 ms)
2025-02-18 02:50:35.780 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888913599 ms)
2025-02-18 02:50:35.786 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:50:25.400211+00:00'} (T+888913605 ms)
2025-02-18 02:51:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+888964417 ms)
2025-02-18 02:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+888964418 ms)
2025-02-18 02:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:51:25.400194+00:00', 'runtime': '888960.001', 'adc_vb_f': '12.668', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888964419 ms)
2025-02-18 02:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:50:25.400207+00:00', 'runtime': '888900.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.668', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+888964419 ms)
2025-02-18 02:51:35.850 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888973669 ms)
2025-02-18 02:51:35.851 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+888973670 ms)
2025-02-18 02:51:35.857 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:51:25.400032+00:00'} (T+888973677 ms)
2025-02-18 02:52:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889024422 ms)
2025-02-18 02:52:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889024423 ms)
2025-02-18 02:52:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:52:25.400176+00:00', 'runtime': '889020.001', 'adc_vb_f': '12.671', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889024424 ms)
2025-02-18 02:52:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:51:25.400194+00:00', 'runtime': '888960.001', 'adc_vb_f': '12.668', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.4', 'ahl_t': '1594.4', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889024424 ms)
2025-02-18 02:52:35.748 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889033568 ms)
2025-02-18 02:52:35.749 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889033568 ms)
2025-02-18 02:52:35.751 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:52:25.405142+00:00'} (T+889033570 ms)
2025-02-18 02:52:55.558 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889053377 ms)
2025-02-18 02:52:55.558 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889053377 ms)
2025-02-18 02:52:55.559 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56720)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889053378 ms)
2025-02-18 02:53:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889084412 ms)
2025-02-18 02:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889084412 ms)
2025-02-18 02:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:53:25.400174+00:00', 'runtime': '889080.001', 'adc_vb_f': '12.668', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889084413 ms)
2025-02-18 02:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:52:25.400176+00:00', 'runtime': '889020.001', 'adc_vb_f': '12.671', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889084414 ms)
2025-02-18 02:53:35.616 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889093435 ms)
2025-02-18 02:53:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889093436 ms)
2025-02-18 02:53:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:53:25.400034+00:00'} (T+889093441 ms)
2025-02-18 02:54:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889144417 ms)
2025-02-18 02:54:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889144417 ms)
2025-02-18 02:54:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:54:25.400209+00:00', 'runtime': '889140.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889144421 ms)
2025-02-18 02:54:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:53:25.400174+00:00', 'runtime': '889080.001', 'adc_vb_f': '12.668', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20133', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889144421 ms)
2025-02-18 02:55:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889204407 ms)
2025-02-18 02:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889204407 ms)
2025-02-18 02:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:55:25.400517+00:00', 'runtime': '889200.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889204411 ms)
2025-02-18 02:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:54:25.400209+00:00', 'runtime': '889140.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.665', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889204412 ms)
2025-02-18 02:56:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889264413 ms)
2025-02-18 02:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889264413 ms)
2025-02-18 02:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:56:25.400174+00:00', 'runtime': '889260.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889264415 ms)
2025-02-18 02:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:55:25.400517+00:00', 'runtime': '889200.001', 'adc_vb_f': '12.665', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889264415 ms)
2025-02-18 02:56:35.899 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889273718 ms)
2025-02-18 02:56:35.899 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889273718 ms)
2025-02-18 02:56:35.899 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56964)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889273719 ms)
2025-02-18 02:57:07.018 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889304837 ms)
2025-02-18 02:57:07.018 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889304838 ms)
2025-02-18 02:57:07.019 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56998)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889304838 ms)
2025-02-18 02:57:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889324402 ms)
2025-02-18 02:57:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889324403 ms)
2025-02-18 02:57:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:57:25.400169+00:00', 'runtime': '889320.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.659', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889324404 ms)
2025-02-18 02:57:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:56:25.400174+00:00', 'runtime': '889260.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889324404 ms)
2025-02-18 02:58:20.166 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889377985 ms)
2025-02-18 02:58:20.166 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889377985 ms)
2025-02-18 02:58:20.167 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57074)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889377986 ms)
2025-02-18 02:58:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889384406 ms)
2025-02-18 02:58:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889384406 ms)
2025-02-18 02:58:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:58:25.400231+00:00', 'runtime': '889380.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889384407 ms)
2025-02-18 02:58:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:57:25.400169+00:00', 'runtime': '889320.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.659', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889384408 ms)
2025-02-18 02:59:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889444411 ms)
2025-02-18 02:59:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889444412 ms)
2025-02-18 02:59:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 02:59:25.400199+00:00', 'runtime': '889440.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.659', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889444413 ms)
2025-02-18 02:59:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:58:25.400231+00:00', 'runtime': '889380.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889444413 ms)
2025-02-18 02:59:36.536 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889454356 ms)
2025-02-18 02:59:36.537 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889454356 ms)
2025-02-18 02:59:36.538 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 02:59:25.400334+00:00'} (T+889454358 ms)
2025-02-18 03:00:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889504416 ms)
2025-02-18 03:00:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889504416 ms)
2025-02-18 03:00:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:00:25.400180+00:00', 'runtime': '889500.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.656', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889504421 ms)
2025-02-18 03:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 02:59:25.400199+00:00', 'runtime': '889440.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.659', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889504421 ms)
2025-02-18 03:00:35.443 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889513263 ms)
2025-02-18 03:00:35.444 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889513263 ms)
2025-02-18 03:00:35.448 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:00:25.400262+00:00'} (T+889513267 ms)
2025-02-18 03:01:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889564408 ms)
2025-02-18 03:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889564408 ms)
2025-02-18 03:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:01:25.400155+00:00', 'runtime': '889560.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889564412 ms)
2025-02-18 03:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:00:25.400180+00:00', 'runtime': '889500.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.656', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.656', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889564413 ms)
2025-02-18 03:01:33.424 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+889571244 ms)
2025-02-18 03:02:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889624413 ms)
2025-02-18 03:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889624413 ms)
2025-02-18 03:02:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:02:25.400161+00:00', 'runtime': '889620.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889624418 ms)
2025-02-18 03:02:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:01:25.400155+00:00', 'runtime': '889560.001', 'adc_vb_f': '12.662', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889624418 ms)
2025-02-18 03:03:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889684403 ms)
2025-02-18 03:03:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889684403 ms)
2025-02-18 03:03:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:03:25.400183+00:00', 'runtime': '889680.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.002', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889684407 ms)
2025-02-18 03:03:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:02:25.400161+00:00', 'runtime': '889620.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889684408 ms)
2025-02-18 03:03:27.595 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889685414 ms)
2025-02-18 03:03:27.595 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889685414 ms)
2025-02-18 03:03:27.595 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57396)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889685415 ms)
2025-02-18 03:03:44.696 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889702515 ms)
2025-02-18 03:03:44.696 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889702516 ms)
2025-02-18 03:03:44.697 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57414)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889702516 ms)
2025-02-18 03:04:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889744410 ms)
2025-02-18 03:04:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889744411 ms)
2025-02-18 03:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:04:25.400191+00:00', 'runtime': '889740.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889744411 ms)
2025-02-18 03:04:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:03:25.400183+00:00', 'runtime': '889680.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.002', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889744412 ms)
2025-02-18 03:05:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889804414 ms)
2025-02-18 03:05:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889804414 ms)
2025-02-18 03:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:05:25.400202+00:00', 'runtime': '889800.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.656', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.65', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889804415 ms)
2025-02-18 03:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:04:25.400191+00:00', 'runtime': '889740.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.662', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.653', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889804415 ms)
2025-02-18 03:06:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889864420 ms)
2025-02-18 03:06:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889864420 ms)
2025-02-18 03:06:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:06:25.400184+00:00', 'runtime': '889860.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.643', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.646', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889864421 ms)
2025-02-18 03:06:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:05:25.400202+00:00', 'runtime': '889800.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.656', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.5', 'ahl_t': '1594.5', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.65', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889864422 ms)
2025-02-18 03:06:36.045 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889873864 ms)
2025-02-18 03:06:36.045 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889873864 ms)
2025-02-18 03:06:36.052 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:06:25.400334+00:00'} (T+889873871 ms)
2025-02-18 03:07:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889924411 ms)
2025-02-18 03:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889924412 ms)
2025-02-18 03:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:07:25.400252+00:00', 'runtime': '889920.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.643', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889924413 ms)
2025-02-18 03:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:06:25.400184+00:00', 'runtime': '889860.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.643', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.646', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889924414 ms)
2025-02-18 03:07:35.623 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889933443 ms)
2025-02-18 03:07:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+889933443 ms)
2025-02-18 03:07:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:07:25.400277+00:00'} (T+889933445 ms)
2025-02-18 03:07:47.064 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889944883 ms)
2025-02-18 03:07:47.064 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889944884 ms)
2025-02-18 03:07:47.065 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57682)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889944884 ms)
2025-02-18 03:07:50.147 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+889947966 ms)
2025-02-18 03:07:50.147 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+889947967 ms)
2025-02-18 03:07:50.148 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57688)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+889947967 ms)
2025-02-18 03:08:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+889984417 ms)
2025-02-18 03:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+889984418 ms)
2025-02-18 03:08:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:08:25.402765+00:00', 'runtime': '889980.003', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.631', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889984422 ms)
2025-02-18 03:08:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:07:25.400252+00:00', 'runtime': '889920.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.643', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+889984422 ms)
2025-02-18 03:09:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890044407 ms)
2025-02-18 03:09:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890044407 ms)
2025-02-18 03:09:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:09:25.400203+00:00', 'runtime': '890040.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.634', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890044408 ms)
2025-02-18 03:09:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:08:25.402765+00:00', 'runtime': '889980.003', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.631', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890044408 ms)
2025-02-18 03:09:35.354 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+890053174 ms)
2025-02-18 03:09:35.355 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+890053174 ms)
2025-02-18 03:09:35.355 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57798)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+890053175 ms)
2025-02-18 03:10:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890104412 ms)
2025-02-18 03:10:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890104413 ms)
2025-02-18 03:10:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:10:25.400172+00:00', 'runtime': '890100.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.653', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890104414 ms)
2025-02-18 03:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:09:25.400203+00:00', 'runtime': '890040.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.634', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890104414 ms)
2025-02-18 03:10:35.928 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890113747 ms)
2025-02-18 03:10:35.929 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890113748 ms)
2025-02-18 03:10:35.933 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:10:25.400313+00:00'} (T+890113753 ms)
2025-02-18 03:11:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890164418 ms)
2025-02-18 03:11:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890164419 ms)
2025-02-18 03:11:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:11:25.400253+00:00', 'runtime': '890160.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.646', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.637', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890164420 ms)
2025-02-18 03:11:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:10:25.400172+00:00', 'runtime': '890100.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.653', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.64', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890164420 ms)
2025-02-18 03:11:35.856 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890173675 ms)
2025-02-18 03:11:35.856 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890173676 ms)
2025-02-18 03:11:35.857 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:11:25.400392+00:00'} (T+890173677 ms)
2025-02-18 03:12:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890224418 ms)
2025-02-18 03:12:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890224419 ms)
2025-02-18 03:12:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:12:25.400222+00:00', 'runtime': '890220.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.643', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890224423 ms)
2025-02-18 03:12:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:11:25.400253+00:00', 'runtime': '890160.001', 'adc_vb_f': '12.659', 'adc_va_f': '1.071', 'adc_vl_f': '12.646', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.659', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.637', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890224423 ms)
2025-02-18 03:12:36.020 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890233839 ms)
2025-02-18 03:12:36.020 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890233839 ms)
2025-02-18 03:12:36.022 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:12:25.400481+00:00'} (T+890233841 ms)
2025-02-18 03:13:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890284409 ms)
2025-02-18 03:13:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890284409 ms)
2025-02-18 03:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:13:25.400158+00:00', 'runtime': '890280.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.646', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890284414 ms)
2025-02-18 03:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:12:25.400222+00:00', 'runtime': '890220.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.643', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890284414 ms)
2025-02-18 03:14:09.756 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+890327575 ms)
2025-02-18 03:14:09.756 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+890327575 ms)
2025-02-18 03:14:09.757 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58092)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+890327576 ms)
2025-02-18 03:14:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890344414 ms)
2025-02-18 03:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890344414 ms)
2025-02-18 03:14:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:14:25.400194+00:00', 'runtime': '890340.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.628', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.653', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890344418 ms)
2025-02-18 03:14:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:13:25.400158+00:00', 'runtime': '890280.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.646', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890344418 ms)
2025-02-18 03:15:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890404419 ms)
2025-02-18 03:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890404419 ms)
2025-02-18 03:15:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:15:25.400173+00:00', 'runtime': '890400.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890404423 ms)
2025-02-18 03:15:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:14:25.400194+00:00', 'runtime': '890340.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.628', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.653', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890404423 ms)
2025-02-18 03:15:35.764 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890413583 ms)
2025-02-18 03:15:35.765 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890413584 ms)
2025-02-18 03:15:35.770 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:15:25.400322+00:00'} (T+890413589 ms)
2025-02-18 03:16:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890464410 ms)
2025-02-18 03:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890464411 ms)
2025-02-18 03:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:16:25.400160+00:00', 'runtime': '890460.001', 'adc_vb_f': '12.653', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890464413 ms)
2025-02-18 03:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:15:25.400173+00:00', 'runtime': '890400.001', 'adc_vb_f': '12.656', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.656', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890464413 ms)
2025-02-18 03:16:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890473428 ms)
2025-02-18 03:16:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890473428 ms)
2025-02-18 03:16:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:16:25.400020+00:00'} (T+890473432 ms)
2025-02-18 03:17:02.038 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+890499857 ms)
2025-02-18 03:17:02.038 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+890499857 ms)
2025-02-18 03:17:02.039 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58286)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+890499858 ms)
2025-02-18 03:17:04.143 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+890501962 ms)
2025-02-18 03:17:04.143 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+890501962 ms)
2025-02-18 03:17:04.143 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58290)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+890501963 ms)
2025-02-18 03:17:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890524416 ms)
2025-02-18 03:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890524416 ms)
2025-02-18 03:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:17:25.400518+00:00', 'runtime': '890520.001', 'adc_vb_f': '12.646', 'adc_va_f': '1.071', 'adc_vl_f': '12.625', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890524418 ms)
2025-02-18 03:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:16:25.400160+00:00', 'runtime': '890460.001', 'adc_vb_f': '12.653', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890524419 ms)
2025-02-18 03:17:35.656 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890533475 ms)
2025-02-18 03:17:35.657 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890533476 ms)
2025-02-18 03:17:35.662 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:17:25.400185+00:00'} (T+890533482 ms)
2025-02-18 03:18:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890584408 ms)
2025-02-18 03:18:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890584408 ms)
2025-02-18 03:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:18:25.400180+00:00', 'runtime': '890580.001', 'adc_vb_f': '12.643', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890584410 ms)
2025-02-18 03:18:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:17:25.400518+00:00', 'runtime': '890520.001', 'adc_vb_f': '12.646', 'adc_va_f': '1.071', 'adc_vl_f': '12.625', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890584410 ms)
2025-02-18 03:19:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890644413 ms)
2025-02-18 03:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890644414 ms)
2025-02-18 03:19:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:19:25.400194+00:00', 'runtime': '890640.001', 'adc_vb_f': '12.643', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.646', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890644417 ms)
2025-02-18 03:19:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:18:25.400180+00:00', 'runtime': '890580.001', 'adc_vb_f': '12.643', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890644418 ms)
2025-02-18 03:20:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890704418 ms)
2025-02-18 03:20:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890704419 ms)
2025-02-18 03:20:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:20:25.400194+00:00', 'runtime': '890700.001', 'adc_vb_f': '12.65', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.646', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890704421 ms)
2025-02-18 03:20:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:19:25.400194+00:00', 'runtime': '890640.001', 'adc_vb_f': '12.643', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.646', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.6', 'ahl_t': '1594.6', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890704421 ms)
2025-02-18 03:20:35.430 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890713249 ms)
2025-02-18 03:20:35.430 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+890713250 ms)
2025-02-18 03:20:35.435 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:20:25.400340+00:00'} (T+890713254 ms)
2025-02-18 03:21:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890764406 ms)
2025-02-18 03:21:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890764407 ms)
2025-02-18 03:21:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:21:25.400167+00:00', 'runtime': '890760.001', 'adc_vb_f': '12.65', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890764410 ms)
2025-02-18 03:21:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:20:25.400194+00:00', 'runtime': '890700.001', 'adc_vb_f': '12.65', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.646', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890764411 ms)
2025-02-18 03:22:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890824412 ms)
2025-02-18 03:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890824413 ms)
2025-02-18 03:22:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:22:25.400264+00:00', 'runtime': '890820.001', 'adc_vb_f': '12.643', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890824417 ms)
2025-02-18 03:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:21:25.400167+00:00', 'runtime': '890760.001', 'adc_vb_f': '12.65', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890824417 ms)
2025-02-18 03:23:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890884404 ms)
2025-02-18 03:23:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890884405 ms)
2025-02-18 03:23:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:23:25.400168+00:00', 'runtime': '890880.001', 'adc_vb_f': '12.653', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890884409 ms)
2025-02-18 03:23:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:22:25.400264+00:00', 'runtime': '890820.001', 'adc_vb_f': '12.643', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890884409 ms)
2025-02-18 03:24:08.758 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+890926577 ms)
2025-02-18 03:24:08.758 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+890926577 ms)
2025-02-18 03:24:08.759 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58738)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+890926578 ms)
2025-02-18 03:24:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+890944410 ms)
2025-02-18 03:24:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+890944411 ms)
2025-02-18 03:24:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:24:25.400200+00:00', 'runtime': '890940.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890944414 ms)
2025-02-18 03:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:23:25.400168+00:00', 'runtime': '890880.001', 'adc_vb_f': '12.653', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+890944414 ms)
2025-02-18 03:24:55.880 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+890973699 ms)
2025-02-18 03:24:55.880 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+890973700 ms)
2025-02-18 03:24:55.881 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58790)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+890973700 ms)
2025-02-18 03:25:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891004415 ms)
2025-02-18 03:25:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891004415 ms)
2025-02-18 03:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:25:25.400188+00:00', 'runtime': '891000.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891004416 ms)
2025-02-18 03:25:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:24:25.400200+00:00', 'runtime': '890940.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.643', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891004417 ms)
2025-02-18 03:25:56.030 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+891033850 ms)
2025-02-18 03:25:56.030 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+891033850 ms)
2025-02-18 03:25:56.031 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58864)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+891033850 ms)
2025-02-18 03:26:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891064404 ms)
2025-02-18 03:26:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891064404 ms)
2025-02-18 03:26:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:26:25.400185+00:00', 'runtime': '891060.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891064405 ms)
2025-02-18 03:26:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:25:25.400188+00:00', 'runtime': '891000.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891064406 ms)
2025-02-18 03:27:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891124409 ms)
2025-02-18 03:27:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891124410 ms)
2025-02-18 03:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:27:25.400184+00:00', 'runtime': '891120.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891124411 ms)
2025-02-18 03:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:26:25.400185+00:00', 'runtime': '891060.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891124411 ms)
2025-02-18 03:27:35.650 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891133469 ms)
2025-02-18 03:27:35.650 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891133470 ms)
2025-02-18 03:27:35.651 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:27:25.400330+00:00'} (T+891133471 ms)
2025-02-18 03:28:08.280 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+891166100 ms)
2025-02-18 03:28:08.280 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+891166100 ms)
2025-02-18 03:28:08.281 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59004)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+891166100 ms)
2025-02-18 03:28:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891184417 ms)
2025-02-18 03:28:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891184417 ms)
2025-02-18 03:28:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:28:25.400175+00:00', 'runtime': '891180.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891184418 ms)
2025-02-18 03:28:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:27:25.400184+00:00', 'runtime': '891120.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.64', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891184419 ms)
2025-02-18 03:29:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891244405 ms)
2025-02-18 03:29:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891244406 ms)
2025-02-18 03:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:29:25.400175+00:00', 'runtime': '891240.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891244407 ms)
2025-02-18 03:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:28:25.400175+00:00', 'runtime': '891180.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891244407 ms)
2025-02-18 03:29:54.416 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+891272235 ms)
2025-02-18 03:29:54.416 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+891272235 ms)
2025-02-18 03:29:54.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+891272236 ms)
2025-02-18 03:29:54.418 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+891272237 ms)
2025-02-18 03:29:54.418 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+891272237 ms)
2025-02-18 03:29:54.419 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+891272238 ms)
2025-02-18 03:30:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891304408 ms)
2025-02-18 03:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891304409 ms)
2025-02-18 03:30:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:30:25.400182+00:00', 'runtime': '891300.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891304410 ms)
2025-02-18 03:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:29:25.400175+00:00', 'runtime': '891240.001', 'adc_vb_f': '12.64', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891304410 ms)
2025-02-18 03:30:35.520 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+891313340 ms)
2025-02-18 03:30:35.521 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+891313340 ms)
2025-02-18 03:30:35.521 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59160)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+891313341 ms)
2025-02-18 03:31:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891364414 ms)
2025-02-18 03:31:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891364415 ms)
2025-02-18 03:31:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:31:25.400619+00:00', 'runtime': '891360.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891364419 ms)
2025-02-18 03:31:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:30:25.400182+00:00', 'runtime': '891300.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891364419 ms)
2025-02-18 03:32:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891424418 ms)
2025-02-18 03:32:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891424418 ms)
2025-02-18 03:32:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:32:25.400222+00:00', 'runtime': '891420.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891424422 ms)
2025-02-18 03:32:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:31:25.400619+00:00', 'runtime': '891360.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891424423 ms)
2025-02-18 03:32:36.072 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891433891 ms)
2025-02-18 03:32:36.072 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891433892 ms)
2025-02-18 03:32:36.074 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:32:25.405067+00:00'} (T+891433893 ms)
2025-02-18 03:33:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891484425 ms)
2025-02-18 03:33:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891484425 ms)
2025-02-18 03:33:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:33:25.400155+00:00', 'runtime': '891480.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891484429 ms)
2025-02-18 03:33:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:32:25.400222+00:00', 'runtime': '891420.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891484430 ms)
2025-02-18 03:34:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891544414 ms)
2025-02-18 03:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891544415 ms)
2025-02-18 03:34:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:34:25.401179+00:00', 'runtime': '891540.002', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891544419 ms)
2025-02-18 03:34:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:33:25.400155+00:00', 'runtime': '891480.001', 'adc_vb_f': '12.637', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.637', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.7', 'ahl_t': '1594.7', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891544419 ms)
2025-02-18 03:35:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891604402 ms)
2025-02-18 03:35:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891604403 ms)
2025-02-18 03:35:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:35:25.400197+00:00', 'runtime': '891600.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891604407 ms)
2025-02-18 03:35:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:34:25.401179+00:00', 'runtime': '891540.002', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891604407 ms)
2025-02-18 03:36:26.606 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891664425 ms)
2025-02-18 03:36:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891664426 ms)
2025-02-18 03:36:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:36:25.400526+00:00', 'runtime': '891660.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891664428 ms)
2025-02-18 03:36:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:35:25.400197+00:00', 'runtime': '891600.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891664428 ms)
2025-02-18 03:36:35.820 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891673639 ms)
2025-02-18 03:36:35.820 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891673640 ms)
2025-02-18 03:36:35.821 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:36:25.401074+00:00'} (T+891673640 ms)
2025-02-18 03:37:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891724415 ms)
2025-02-18 03:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891724415 ms)
2025-02-18 03:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:37:25.402579+00:00', 'runtime': '891720.003', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891724417 ms)
2025-02-18 03:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:36:25.400526+00:00', 'runtime': '891660.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891724418 ms)
2025-02-18 03:37:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891733445 ms)
2025-02-18 03:37:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891733446 ms)
2025-02-18 03:37:35.629 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:37:25.400005+00:00'} (T+891733449 ms)
2025-02-18 03:38:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891784421 ms)
2025-02-18 03:38:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891784421 ms)
2025-02-18 03:38:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:38:25.400149+00:00', 'runtime': '891780.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891784423 ms)
2025-02-18 03:38:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:37:25.402579+00:00', 'runtime': '891720.003', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891784423 ms)
2025-02-18 03:38:35.627 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891793446 ms)
2025-02-18 03:38:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891793447 ms)
2025-02-18 03:38:35.629 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:38:25.401140+00:00'} (T+891793448 ms)
2025-02-18 03:39:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891844410 ms)
2025-02-18 03:39:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891844410 ms)
2025-02-18 03:39:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:39:25.402796+00:00', 'runtime': '891840.003', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891844414 ms)
2025-02-18 03:39:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:38:25.400149+00:00', 'runtime': '891780.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891844415 ms)
2025-02-18 03:39:35.463 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891853283 ms)
2025-02-18 03:39:35.464 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891853283 ms)
2025-02-18 03:39:35.465 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:39:25.400183+00:00'} (T+891853284 ms)
2025-02-18 03:40:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891904415 ms)
2025-02-18 03:40:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891904416 ms)
2025-02-18 03:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:40:25.400210+00:00', 'runtime': '891900.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891904420 ms)
2025-02-18 03:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:39:25.402796+00:00', 'runtime': '891840.003', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891904420 ms)
2025-02-18 03:40:37.011 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891914830 ms)
2025-02-18 03:40:37.012 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+891914831 ms)
2025-02-18 03:40:37.013 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:40:25.413125+00:00'} (T+891914832 ms)
2025-02-18 03:40:47.370 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+891925189 ms)
2025-02-18 03:40:47.370 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+891925189 ms)
2025-02-18 03:40:47.370 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59816)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+891925190 ms)
2025-02-18 03:41:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+891964424 ms)
2025-02-18 03:41:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+891964424 ms)
2025-02-18 03:41:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:41:25.400178+00:00', 'runtime': '891960.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891964426 ms)
2025-02-18 03:41:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:40:25.400210+00:00', 'runtime': '891900.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+891964426 ms)
2025-02-18 03:42:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892024415 ms)
2025-02-18 03:42:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892024415 ms)
2025-02-18 03:42:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:42:25.400669+00:00', 'runtime': '892020.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892024416 ms)
2025-02-18 03:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:41:25.400178+00:00', 'runtime': '891960.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892024417 ms)
2025-02-18 03:43:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892084402 ms)
2025-02-18 03:43:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892084402 ms)
2025-02-18 03:43:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:43:25.400168+00:00', 'runtime': '892080.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892084403 ms)
2025-02-18 03:43:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:42:25.400669+00:00', 'runtime': '892020.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892084404 ms)
2025-02-18 03:43:32.676 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+892090496 ms)
2025-02-18 03:43:32.677 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+892090496 ms)
2025-02-18 03:43:32.677 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59990)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892090497 ms)
2025-02-18 03:43:41.743 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+892099563 ms)
2025-02-18 03:43:41.744 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+892099563 ms)
2025-02-18 03:43:41.744 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60000)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892099563 ms)
2025-02-18 03:44:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892144409 ms)
2025-02-18 03:44:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892144410 ms)
2025-02-18 03:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:44:25.400174+00:00', 'runtime': '892140.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892144411 ms)
2025-02-18 03:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:43:25.400168+00:00', 'runtime': '892080.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892144411 ms)
2025-02-18 03:44:41.892 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+892159711 ms)
2025-02-18 03:44:41.892 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+892159711 ms)
2025-02-18 03:44:41.893 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60066)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892159712 ms)
2025-02-18 03:45:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892204413 ms)
2025-02-18 03:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892204414 ms)
2025-02-18 03:45:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:45:25.400170+00:00', 'runtime': '892200.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892204415 ms)
2025-02-18 03:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:44:25.400174+00:00', 'runtime': '892140.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892204415 ms)
2025-02-18 03:46:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892264418 ms)
2025-02-18 03:46:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892264418 ms)
2025-02-18 03:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:46:25.400172+00:00', 'runtime': '892260.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892264419 ms)
2025-02-18 03:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:45:25.400170+00:00', 'runtime': '892200.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892264419 ms)
2025-02-18 03:46:35.729 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892273549 ms)
2025-02-18 03:46:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892273549 ms)
2025-02-18 03:46:35.731 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:46:25.400687+00:00'} (T+892273550 ms)
2025-02-18 03:47:25.684 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892323503 ms)
2025-02-18 03:47:25.684 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892323504 ms)
2025-02-18 03:47:25.685 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:47:25.400192+00:00', 'runtime': '892320.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892323505 ms)
2025-02-18 03:47:25.686 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:46:25.400172+00:00', 'runtime': '892260.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892323505 ms)
2025-02-18 03:47:51.172 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+892348991 ms)
2025-02-18 03:47:51.172 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+892348992 ms)
2025-02-18 03:47:51.173 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60278)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892348992 ms)
2025-02-18 03:48:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892384411 ms)
2025-02-18 03:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892384412 ms)
2025-02-18 03:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:48:25.400075+00:00', 'runtime': '892380.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892384413 ms)
2025-02-18 03:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:47:25.400192+00:00', 'runtime': '892320.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.8', 'ahl_t': '1594.8', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892384413 ms)
2025-02-18 03:49:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892444414 ms)
2025-02-18 03:49:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892444415 ms)
2025-02-18 03:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:49:25.400197+00:00', 'runtime': '892440.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892444416 ms)
2025-02-18 03:49:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:48:25.400075+00:00', 'runtime': '892380.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892444416 ms)
2025-02-18 03:49:35.808 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892453628 ms)
2025-02-18 03:49:35.809 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892453628 ms)
2025-02-18 03:49:35.810 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:49:25.400334+00:00'} (T+892453630 ms)
2025-02-18 03:50:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892504405 ms)
2025-02-18 03:50:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892504405 ms)
2025-02-18 03:50:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:50:25.400199+00:00', 'runtime': '892500.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892504406 ms)
2025-02-18 03:50:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:49:25.400197+00:00', 'runtime': '892440.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892504407 ms)
2025-02-18 03:50:35.859 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892513678 ms)
2025-02-18 03:50:35.860 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892513679 ms)
2025-02-18 03:50:35.860 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:50:25.400010+00:00'} (T+892513680 ms)
2025-02-18 03:51:15.498 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+892553317 ms)
2025-02-18 03:51:15.498 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+892553318 ms)
2025-02-18 03:51:15.499 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60494)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892553318 ms)
2025-02-18 03:51:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892564410 ms)
2025-02-18 03:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892564410 ms)
2025-02-18 03:51:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:51:25.400202+00:00', 'runtime': '892560.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892564411 ms)
2025-02-18 03:51:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:50:25.400199+00:00', 'runtime': '892500.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892564412 ms)
2025-02-18 03:51:35.763 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892573582 ms)
2025-02-18 03:51:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892573583 ms)
2025-02-18 03:51:35.765 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:51:25.400471+00:00'} (T+892573584 ms)
2025-02-18 03:52:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892624415 ms)
2025-02-18 03:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892624416 ms)
2025-02-18 03:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:52:25.400173+00:00', 'runtime': '892620.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892624417 ms)
2025-02-18 03:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:51:25.400202+00:00', 'runtime': '892560.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892624417 ms)
2025-02-18 03:52:35.730 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892633549 ms)
2025-02-18 03:52:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892633550 ms)
2025-02-18 03:52:35.731 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:52:25.400037+00:00'} (T+892633551 ms)
2025-02-18 03:52:47.609 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+892645429 ms)
2025-02-18 03:52:47.610 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+892645429 ms)
2025-02-18 03:52:47.610 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892645430 ms)
2025-02-18 03:52:47.611 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+892645431 ms)
2025-02-18 03:52:47.612 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+892645431 ms)
2025-02-18 03:52:47.612 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892645431 ms)
2025-02-18 03:52:47.613 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+892645433 ms)
2025-02-18 03:52:47.614 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+892645433 ms)
2025-02-18 03:52:47.614 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892645433 ms)
2025-02-18 03:53:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892684406 ms)
2025-02-18 03:53:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892684406 ms)
2025-02-18 03:53:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:53:25.400174+00:00', 'runtime': '892680.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892684407 ms)
2025-02-18 03:53:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:52:25.400173+00:00', 'runtime': '892620.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892684408 ms)
2025-02-18 03:53:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892693423 ms)
2025-02-18 03:53:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892693424 ms)
2025-02-18 03:53:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:53:25.400054+00:00'} (T+892693425 ms)
2025-02-18 03:54:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892744414 ms)
2025-02-18 03:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892744414 ms)
2025-02-18 03:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:54:25.400182+00:00', 'runtime': '892740.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892744415 ms)
2025-02-18 03:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:53:25.400174+00:00', 'runtime': '892680.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20134', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892744416 ms)
2025-02-18 03:55:03.851 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+892781670 ms)
2025-02-18 03:55:03.851 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+892781671 ms)
2025-02-18 03:55:03.852 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60740)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892781671 ms)
2025-02-18 03:55:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892804418 ms)
2025-02-18 03:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892804418 ms)
2025-02-18 03:55:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:55:25.400209+00:00', 'runtime': '892800.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892804419 ms)
2025-02-18 03:55:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:54:25.400182+00:00', 'runtime': '892740.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892804420 ms)
2025-02-18 03:56:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892864405 ms)
2025-02-18 03:56:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892864405 ms)
2025-02-18 03:56:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:56:25.400207+00:00', 'runtime': '892860.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892864406 ms)
2025-02-18 03:56:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:55:25.400209+00:00', 'runtime': '892800.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892864407 ms)
2025-02-18 03:57:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892924410 ms)
2025-02-18 03:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892924411 ms)
2025-02-18 03:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:57:25.400176+00:00', 'runtime': '892920.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892924412 ms)
2025-02-18 03:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:56:25.400207+00:00', 'runtime': '892860.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892924412 ms)
2025-02-18 03:57:35.427 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892933246 ms)
2025-02-18 03:57:35.427 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+892933247 ms)
2025-02-18 03:57:35.428 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 03:57:25.400339+00:00'} (T+892933248 ms)
2025-02-18 03:58:09.190 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+892967009 ms)
2025-02-18 03:58:09.190 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+892967010 ms)
2025-02-18 03:58:09.191 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60944)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+892967010 ms)
2025-02-18 03:58:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+892984416 ms)
2025-02-18 03:58:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+892984416 ms)
2025-02-18 03:58:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:58:25.400257+00:00', 'runtime': '892980.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892984417 ms)
2025-02-18 03:58:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:57:25.400176+00:00', 'runtime': '892920.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+892984418 ms)
2025-02-18 03:59:15.274 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+893033093 ms)
2025-02-18 03:59:15.274 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+893033093 ms)
2025-02-18 03:59:15.275 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893033094 ms)
2025-02-18 03:59:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893044405 ms)
2025-02-18 03:59:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893044405 ms)
2025-02-18 03:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 03:59:25.400163+00:00', 'runtime': '893040.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893044406 ms)
2025-02-18 03:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:58:25.400257+00:00', 'runtime': '892980.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893044407 ms)
2025-02-18 04:00:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893104412 ms)
2025-02-18 04:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893104413 ms)
2025-02-18 04:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:00:25.400184+00:00', 'runtime': '893100.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893104414 ms)
2025-02-18 04:00:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 03:59:25.400163+00:00', 'runtime': '893040.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893104414 ms)
2025-02-18 04:01:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893164418 ms)
2025-02-18 04:01:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893164418 ms)
2025-02-18 04:01:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:01:25.400170+00:00', 'runtime': '893160.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893164419 ms)
2025-02-18 04:01:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:00:25.400184+00:00', 'runtime': '893100.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1594.9', 'ahl_t': '1594.9', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893164420 ms)
2025-02-18 04:02:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893224407 ms)
2025-02-18 04:02:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893224408 ms)
2025-02-18 04:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:02:25.400197+00:00', 'runtime': '893220.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893224409 ms)
2025-02-18 04:02:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:01:25.400170+00:00', 'runtime': '893160.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893224409 ms)
2025-02-18 04:03:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893284413 ms)
2025-02-18 04:03:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893284413 ms)
2025-02-18 04:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:03:25.400175+00:00', 'runtime': '893280.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893284414 ms)
2025-02-18 04:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:02:25.400197+00:00', 'runtime': '893220.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893284415 ms)
2025-02-18 04:04:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893344406 ms)
2025-02-18 04:04:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893344406 ms)
2025-02-18 04:04:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:04:25.400159+00:00', 'runtime': '893340.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893344407 ms)
2025-02-18 04:04:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:03:25.400175+00:00', 'runtime': '893280.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893344408 ms)
2025-02-18 04:04:35.727 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893353547 ms)
2025-02-18 04:04:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893353547 ms)
2025-02-18 04:04:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:04:25.400322+00:00'} (T+893353548 ms)
2025-02-18 04:05:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893404410 ms)
2025-02-18 04:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893404410 ms)
2025-02-18 04:05:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:05:25.400206+00:00', 'runtime': '893400.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893404411 ms)
2025-02-18 04:05:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:04:25.400159+00:00', 'runtime': '893340.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893404412 ms)
2025-02-18 04:05:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893413429 ms)
2025-02-18 04:05:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893413430 ms)
2025-02-18 04:05:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:05:25.400165+00:00'} (T+893413431 ms)
2025-02-18 04:06:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893464415 ms)
2025-02-18 04:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893464415 ms)
2025-02-18 04:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:06:25.400189+00:00', 'runtime': '893460.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893464416 ms)
2025-02-18 04:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:05:25.400206+00:00', 'runtime': '893400.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893464417 ms)
2025-02-18 04:07:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893524421 ms)
2025-02-18 04:07:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893524421 ms)
2025-02-18 04:07:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:07:25.401134+00:00', 'runtime': '893520.002', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893524423 ms)
2025-02-18 04:07:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:06:25.400189+00:00', 'runtime': '893460.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893524424 ms)
2025-02-18 04:08:05.019 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+893562838 ms)
2025-02-18 04:08:05.019 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+893562838 ms)
2025-02-18 04:08:05.020 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33348)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893562839 ms)
2025-02-18 04:08:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893584412 ms)
2025-02-18 04:08:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893584413 ms)
2025-02-18 04:08:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:08:25.400609+00:00', 'runtime': '893580.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893584415 ms)
2025-02-18 04:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:07:25.401134+00:00', 'runtime': '893520.002', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893584415 ms)
2025-02-18 04:09:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893644418 ms)
2025-02-18 04:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893644419 ms)
2025-02-18 04:09:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:09:25.402775+00:00', 'runtime': '893640.003', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893644420 ms)
2025-02-18 04:09:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:08:25.400609+00:00', 'runtime': '893580.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893644421 ms)
2025-02-18 04:09:33.130 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+893650949 ms)
2025-02-18 04:09:33.130 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+893650949 ms)
2025-02-18 04:09:33.131 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893650951 ms)
2025-02-18 04:09:33.134 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+893650953 ms)
2025-02-18 04:09:33.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+893650953 ms)
2025-02-18 04:09:33.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893650954 ms)
2025-02-18 04:09:33.137 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+893650956 ms)
2025-02-18 04:09:33.137 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+893650957 ms)
2025-02-18 04:09:33.139 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893650958 ms)
2025-02-18 04:09:35.660 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893653479 ms)
2025-02-18 04:09:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893653480 ms)
2025-02-18 04:09:35.661 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:09:25.400133+00:00'} (T+893653481 ms)
2025-02-18 04:10:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893704423 ms)
2025-02-18 04:10:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893704423 ms)
2025-02-18 04:10:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:10:25.400136+00:00', 'runtime': '893700.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893704424 ms)
2025-02-18 04:10:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:09:25.402775+00:00', 'runtime': '893640.003', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.634', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893704425 ms)
2025-02-18 04:10:29.280 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+893707099 ms)
2025-02-18 04:10:29.280 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+893707099 ms)
2025-02-18 04:10:29.281 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33500)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893707100 ms)
2025-02-18 04:11:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893764413 ms)
2025-02-18 04:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893764414 ms)
2025-02-18 04:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:11:25.400627+00:00', 'runtime': '893760.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893764415 ms)
2025-02-18 04:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:10:25.400136+00:00', 'runtime': '893700.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893764415 ms)
2025-02-18 04:12:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893824416 ms)
2025-02-18 04:12:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893824416 ms)
2025-02-18 04:12:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:12:25.400544+00:00', 'runtime': '893820.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893824417 ms)
2025-02-18 04:12:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:11:25.400627+00:00', 'runtime': '893760.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893824418 ms)
2025-02-18 04:12:57.518 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+893855338 ms)
2025-02-18 04:12:57.519 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+893855338 ms)
2025-02-18 04:12:57.519 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33658)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893855338 ms)
2025-02-18 04:12:59.590 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+893857409 ms)
2025-02-18 04:12:59.590 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+893857410 ms)
2025-02-18 04:12:59.591 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33662)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+893857410 ms)
2025-02-18 04:13:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893884421 ms)
2025-02-18 04:13:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893884422 ms)
2025-02-18 04:13:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:13:25.400185+00:00', 'runtime': '893880.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893884424 ms)
2025-02-18 04:13:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:12:25.400544+00:00', 'runtime': '893820.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893884424 ms)
2025-02-18 04:13:35.881 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893893700 ms)
2025-02-18 04:13:35.881 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893893701 ms)
2025-02-18 04:13:35.882 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:13:25.413049+00:00'} (T+893893702 ms)
2025-02-18 04:14:25.580 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+893943399 ms)
2025-02-18 04:14:25.580 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+893943399 ms)
2025-02-18 04:14:25.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:14:25.400150+00:00', 'runtime': '893940.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893943404 ms)
2025-02-18 04:14:25.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:13:25.400185+00:00', 'runtime': '893880.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+893943404 ms)
2025-02-18 04:14:37.007 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893954826 ms)
2025-02-18 04:14:37.008 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+893954827 ms)
2025-02-18 04:14:37.009 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:14:25.413124+00:00'} (T+893954828 ms)
2025-02-18 04:15:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894004416 ms)
2025-02-18 04:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894004417 ms)
2025-02-18 04:15:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:15:25.400170+00:00', 'runtime': '894000.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.628', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894004421 ms)
2025-02-18 04:15:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:14:25.400150+00:00', 'runtime': '893940.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.0', 'ahl_t': '1595.0', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.631', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894004421 ms)
2025-02-18 04:15:26.848 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+894004667 ms)
2025-02-18 04:15:26.848 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+894004667 ms)
2025-02-18 04:15:26.848 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33818)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+894004668 ms)
2025-02-18 04:16:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894064406 ms)
2025-02-18 04:16:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894064406 ms)
2025-02-18 04:16:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:16:25.400795+00:00', 'runtime': '894060.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894064408 ms)
2025-02-18 04:16:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:15:25.400170+00:00', 'runtime': '894000.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.628', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894064409 ms)
2025-02-18 04:17:07.049 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+894104868 ms)
2025-02-18 04:17:07.049 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+894104868 ms)
2025-02-18 04:17:07.049 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33936)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+894104869 ms)
2025-02-18 04:17:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894124411 ms)
2025-02-18 04:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894124411 ms)
2025-02-18 04:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:17:25.400188+00:00', 'runtime': '894120.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894124413 ms)
2025-02-18 04:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:16:25.400795+00:00', 'runtime': '894060.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894124413 ms)
2025-02-18 04:18:23.220 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+894181039 ms)
2025-02-18 04:18:23.220 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+894181039 ms)
2025-02-18 04:18:23.220 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34016)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+894181040 ms)
2025-02-18 04:18:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894184419 ms)
2025-02-18 04:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894184419 ms)
2025-02-18 04:18:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:18:25.400194+00:00', 'runtime': '894180.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894184421 ms)
2025-02-18 04:18:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:17:25.400188+00:00', 'runtime': '894120.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894184422 ms)
2025-02-18 04:19:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894244409 ms)
2025-02-18 04:19:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894244410 ms)
2025-02-18 04:19:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:19:25.400189+00:00', 'runtime': '894240.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894244412 ms)
2025-02-18 04:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:18:25.400194+00:00', 'runtime': '894180.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894244412 ms)
2025-02-18 04:20:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894304403 ms)
2025-02-18 04:20:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894304403 ms)
2025-02-18 04:20:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:20:25.400636+00:00', 'runtime': '894300.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894304405 ms)
2025-02-18 04:20:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:19:25.400189+00:00', 'runtime': '894240.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894304406 ms)
2025-02-18 04:21:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894364411 ms)
2025-02-18 04:21:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894364411 ms)
2025-02-18 04:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:21:25.400193+00:00', 'runtime': '894360.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894364413 ms)
2025-02-18 04:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:20:25.400636+00:00', 'runtime': '894300.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894364413 ms)
2025-02-18 04:22:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894424418 ms)
2025-02-18 04:22:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894424418 ms)
2025-02-18 04:22:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:22:25.400558+00:00', 'runtime': '894420.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894424420 ms)
2025-02-18 04:22:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:21:25.400193+00:00', 'runtime': '894360.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894424420 ms)
2025-02-18 04:23:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894484410 ms)
2025-02-18 04:23:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894484411 ms)
2025-02-18 04:23:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:23:25.400178+00:00', 'runtime': '894480.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894484413 ms)
2025-02-18 04:23:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:22:25.400558+00:00', 'runtime': '894420.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894484413 ms)
2025-02-18 04:24:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894544418 ms)
2025-02-18 04:24:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894544419 ms)
2025-02-18 04:24:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:24:25.400200+00:00', 'runtime': '894540.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894544421 ms)
2025-02-18 04:24:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:23:25.400178+00:00', 'runtime': '894480.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894544421 ms)
2025-02-18 04:25:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894604406 ms)
2025-02-18 04:25:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894604406 ms)
2025-02-18 04:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:25:25.400806+00:00', 'runtime': '894600.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894604408 ms)
2025-02-18 04:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:24:25.400200+00:00', 'runtime': '894540.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894604409 ms)
2025-02-18 04:25:30.853 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+894608672 ms)
2025-02-18 04:25:30.853 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+894608672 ms)
2025-02-18 04:25:30.854 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34464)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+894608673 ms)
2025-02-18 04:26:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894664412 ms)
2025-02-18 04:26:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894664413 ms)
2025-02-18 04:26:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:26:25.400650+00:00', 'runtime': '894660.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894664414 ms)
2025-02-18 04:26:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:25:25.400806+00:00', 'runtime': '894600.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894664414 ms)
2025-02-18 04:27:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894724403 ms)
2025-02-18 04:27:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894724403 ms)
2025-02-18 04:27:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:27:25.400179+00:00', 'runtime': '894720.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894724404 ms)
2025-02-18 04:27:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:26:25.400650+00:00', 'runtime': '894660.001', 'adc_vb_f': '12.634', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894724404 ms)
2025-02-18 04:27:35.652 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+894733472 ms)
2025-02-18 04:27:35.653 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+894733472 ms)
2025-02-18 04:27:35.654 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:27:25.400346+00:00'} (T+894733473 ms)
2025-02-18 04:27:50.125 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+894747945 ms)
2025-02-18 04:27:50.126 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+894747945 ms)
2025-02-18 04:27:50.126 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34622)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+894747945 ms)
2025-02-18 04:28:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894784411 ms)
2025-02-18 04:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894784411 ms)
2025-02-18 04:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:28:25.400190+00:00', 'runtime': '894780.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894784412 ms)
2025-02-18 04:28:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:27:25.400179+00:00', 'runtime': '894720.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894784413 ms)
2025-02-18 04:28:35.691 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+894793510 ms)
2025-02-18 04:28:35.691 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+894793510 ms)
2025-02-18 04:28:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:28:25.400157+00:00'} (T+894793511 ms)
2025-02-18 04:29:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894844416 ms)
2025-02-18 04:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894844417 ms)
2025-02-18 04:29:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:29:25.400213+00:00', 'runtime': '894840.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894844418 ms)
2025-02-18 04:29:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:28:25.400190+00:00', 'runtime': '894780.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.1', 'ahl_t': '1595.1', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894844418 ms)
2025-02-18 04:29:35.747 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+894853566 ms)
2025-02-18 04:29:35.748 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+894853567 ms)
2025-02-18 04:29:35.749 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:29:25.400138+00:00'} (T+894853568 ms)
2025-02-18 04:30:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894904406 ms)
2025-02-18 04:30:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894904406 ms)
2025-02-18 04:30:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:30:25.400161+00:00', 'runtime': '894900.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894904407 ms)
2025-02-18 04:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:29:25.400213+00:00', 'runtime': '894840.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894904408 ms)
2025-02-18 04:31:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+894964412 ms)
2025-02-18 04:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+894964413 ms)
2025-02-18 04:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:31:25.400071+00:00', 'runtime': '894960.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894964415 ms)
2025-02-18 04:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:30:25.400161+00:00', 'runtime': '894900.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+894964415 ms)
2025-02-18 04:32:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895024419 ms)
2025-02-18 04:32:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895024419 ms)
2025-02-18 04:32:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:32:25.400222+00:00', 'runtime': '895020.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895024421 ms)
2025-02-18 04:32:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:31:25.400071+00:00', 'runtime': '894960.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895024422 ms)
2025-02-18 04:32:35.745 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895033565 ms)
2025-02-18 04:32:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895033565 ms)
2025-02-18 04:32:35.747 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:32:25.413053+00:00'} (T+895033566 ms)
2025-02-18 04:33:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895084409 ms)
2025-02-18 04:33:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895084409 ms)
2025-02-18 04:33:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:33:25.402705+00:00', 'runtime': '895080.003', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895084413 ms)
2025-02-18 04:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:32:25.400222+00:00', 'runtime': '895020.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895084414 ms)
2025-02-18 04:34:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895144403 ms)
2025-02-18 04:34:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895144403 ms)
2025-02-18 04:34:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:34:25.400820+00:00', 'runtime': '895140.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.616', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895144407 ms)
2025-02-18 04:34:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:33:25.402705+00:00', 'runtime': '895080.003', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.619', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895144408 ms)
2025-02-18 04:34:29.701 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+895147521 ms)
2025-02-18 04:34:29.702 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+895147521 ms)
2025-02-18 04:34:29.702 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35044)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+895147521 ms)
2025-02-18 04:35:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895204422 ms)
2025-02-18 04:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895204422 ms)
2025-02-18 04:35:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:35:25.400260+00:00', 'runtime': '895200.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.616', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895204426 ms)
2025-02-18 04:35:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:34:25.400820+00:00', 'runtime': '895140.001', 'adc_vb_f': '12.631', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.616', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895204427 ms)
2025-02-18 04:35:59.902 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+895237721 ms)
2025-02-18 04:35:59.902 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+895237722 ms)
2025-02-18 04:35:59.903 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35152)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+895237722 ms)
2025-02-18 04:36:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895264415 ms)
2025-02-18 04:36:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895264416 ms)
2025-02-18 04:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:36:25.400158+00:00', 'runtime': '895260.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.619', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895264417 ms)
2025-02-18 04:36:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:35:25.400260+00:00', 'runtime': '895200.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.628', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.616', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895264417 ms)
2025-02-18 04:37:26.094 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+895323913 ms)
2025-02-18 04:37:26.094 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+895323913 ms)
2025-02-18 04:37:26.094 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35242)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+895323913 ms)
2025-02-18 04:37:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895324403 ms)
2025-02-18 04:37:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895324404 ms)
2025-02-18 04:37:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:37:25.400167+00:00', 'runtime': '895320.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895324405 ms)
2025-02-18 04:37:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:36:25.400158+00:00', 'runtime': '895260.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.619', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895324405 ms)
2025-02-18 04:38:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895384411 ms)
2025-02-18 04:38:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895384412 ms)
2025-02-18 04:38:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:38:25.400182+00:00', 'runtime': '895380.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895384413 ms)
2025-02-18 04:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:37:25.400167+00:00', 'runtime': '895320.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895384413 ms)
2025-02-18 04:38:35.685 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895393504 ms)
2025-02-18 04:38:35.686 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895393505 ms)
2025-02-18 04:38:35.687 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:38:25.400337+00:00'} (T+895393506 ms)
2025-02-18 04:39:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895444417 ms)
2025-02-18 04:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895444418 ms)
2025-02-18 04:39:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:39:25.400188+00:00', 'runtime': '895440.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895444419 ms)
2025-02-18 04:39:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:38:25.400182+00:00', 'runtime': '895380.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895444419 ms)
2025-02-18 04:39:35.626 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895453445 ms)
2025-02-18 04:39:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895453446 ms)
2025-02-18 04:39:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:39:25.400135+00:00'} (T+895453447 ms)
2025-02-18 04:40:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895504409 ms)
2025-02-18 04:40:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895504410 ms)
2025-02-18 04:40:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:40:25.400207+00:00', 'runtime': '895500.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895504411 ms)
2025-02-18 04:40:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:39:25.400188+00:00', 'runtime': '895440.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.607', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895504411 ms)
2025-02-18 04:40:35.693 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895513513 ms)
2025-02-18 04:40:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895513513 ms)
2025-02-18 04:40:35.695 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:40:25.400209+00:00'} (T+895513514 ms)
2025-02-18 04:41:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895564415 ms)
2025-02-18 04:41:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895564416 ms)
2025-02-18 04:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:41:25.400180+00:00', 'runtime': '895560.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895564417 ms)
2025-02-18 04:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:40:25.400207+00:00', 'runtime': '895500.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895564417 ms)
2025-02-18 04:41:58.505 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+895596324 ms)
2025-02-18 04:41:58.505 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+895596325 ms)
2025-02-18 04:41:58.506 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35534)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+895596325 ms)
2025-02-18 04:42:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895624407 ms)
2025-02-18 04:42:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895624408 ms)
2025-02-18 04:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:42:25.400166+00:00', 'runtime': '895620.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895624409 ms)
2025-02-18 04:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:41:25.400180+00:00', 'runtime': '895560.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895624409 ms)
2025-02-18 04:43:12.597 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+895670417 ms)
2025-02-18 04:43:12.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+895670417 ms)
2025-02-18 04:43:12.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+895670417 ms)
2025-02-18 04:43:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895684413 ms)
2025-02-18 04:43:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895684414 ms)
2025-02-18 04:43:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:43:25.400194+00:00', 'runtime': '895680.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895684415 ms)
2025-02-18 04:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:42:25.400166+00:00', 'runtime': '895620.001', 'adc_vb_f': '12.625', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.2', 'ahl_t': '1595.2', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '1.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895684415 ms)
2025-02-18 04:44:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895744403 ms)
2025-02-18 04:44:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895744404 ms)
2025-02-18 04:44:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:44:25.400197+00:00', 'runtime': '895740.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.619', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895744405 ms)
2025-02-18 04:44:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:43:25.400194+00:00', 'runtime': '895680.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895744405 ms)
2025-02-18 04:45:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895804410 ms)
2025-02-18 04:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895804410 ms)
2025-02-18 04:45:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:45:25.400187+00:00', 'runtime': '895800.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895804411 ms)
2025-02-18 04:45:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:44:25.400197+00:00', 'runtime': '895740.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.619', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895804412 ms)
2025-02-18 04:46:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895864415 ms)
2025-02-18 04:46:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895864415 ms)
2025-02-18 04:46:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:46:25.400168+00:00', 'runtime': '895860.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.619', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895864416 ms)
2025-02-18 04:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:45:25.400187+00:00', 'runtime': '895800.001', 'adc_vb_f': '12.622', 'adc_va_f': '1.071', 'adc_vl_f': '12.622', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.622', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.604', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895864417 ms)
2025-02-18 04:46:35.952 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895873772 ms)
2025-02-18 04:46:35.953 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895873772 ms)
2025-02-18 04:46:35.959 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:46:25.400346+00:00'} (T+895873778 ms)
2025-02-18 04:47:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895924405 ms)
2025-02-18 04:47:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895924405 ms)
2025-02-18 04:47:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:47:25.400207+00:00', 'runtime': '895920.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.613', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.619', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895924406 ms)
2025-02-18 04:47:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:46:25.400168+00:00', 'runtime': '895860.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.619', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895924407 ms)
2025-02-18 04:47:35.839 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895933659 ms)
2025-02-18 04:47:35.840 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895933659 ms)
2025-02-18 04:47:35.845 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:47:25.400363+00:00'} (T+895933664 ms)
2025-02-18 04:48:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+895984411 ms)
2025-02-18 04:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+895984411 ms)
2025-02-18 04:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:48:25.400179+00:00', 'runtime': '895980.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.607', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.619', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895984412 ms)
2025-02-18 04:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:47:25.400207+00:00', 'runtime': '895920.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.613', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.619', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+895984413 ms)
2025-02-18 04:48:32.099 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+895989918 ms)
2025-02-18 04:48:32.099 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+895989919 ms)
2025-02-18 04:48:32.100 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35956)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+895989919 ms)
2025-02-18 04:48:35.626 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895993445 ms)
2025-02-18 04:48:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+895993446 ms)
2025-02-18 04:48:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:48:25.400473+00:00'} (T+895993450 ms)
2025-02-18 04:49:19.229 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+896037048 ms)
2025-02-18 04:49:19.229 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+896037048 ms)
2025-02-18 04:49:19.229 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36008)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896037049 ms)
2025-02-18 04:49:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896044415 ms)
2025-02-18 04:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896044416 ms)
2025-02-18 04:49:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:49:25.400226+00:00', 'runtime': '896040.001', 'adc_vb_f': '12.613', 'adc_va_f': '1.071', 'adc_vl_f': '12.595', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896044417 ms)
2025-02-18 04:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:48:25.400179+00:00', 'runtime': '895980.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.607', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.619', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896044417 ms)
2025-02-18 04:49:35.746 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896053565 ms)
2025-02-18 04:49:35.746 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896053565 ms)
2025-02-18 04:49:35.747 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:49:25.400216+00:00'} (T+896053567 ms)
2025-02-18 04:50:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896104419 ms)
2025-02-18 04:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896104419 ms)
2025-02-18 04:50:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:50:25.402689+00:00', 'runtime': '896100.003', 'adc_vb_f': '12.616', 'adc_va_f': '1.071', 'adc_vl_f': '12.601', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896104420 ms)
2025-02-18 04:50:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:49:25.400226+00:00', 'runtime': '896040.001', 'adc_vb_f': '12.613', 'adc_va_f': '1.071', 'adc_vl_f': '12.595', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896104421 ms)
2025-02-18 04:50:32.417 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+896110236 ms)
2025-02-18 04:50:32.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+896110237 ms)
2025-02-18 04:50:32.418 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36086)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896110237 ms)
2025-02-18 04:50:35.759 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896113578 ms)
2025-02-18 04:50:35.760 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896113579 ms)
2025-02-18 04:50:35.761 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:50:25.400099+00:00'} (T+896113580 ms)
2025-02-18 04:51:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896164408 ms)
2025-02-18 04:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896164409 ms)
2025-02-18 04:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:51:25.400059+00:00', 'runtime': '896160.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896164410 ms)
2025-02-18 04:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:50:25.402689+00:00', 'runtime': '896100.003', 'adc_vb_f': '12.616', 'adc_va_f': '1.071', 'adc_vl_f': '12.601', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896164410 ms)
2025-02-18 04:52:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896224413 ms)
2025-02-18 04:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896224413 ms)
2025-02-18 04:52:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:52:25.400588+00:00', 'runtime': '896220.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896224414 ms)
2025-02-18 04:52:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:51:25.400059+00:00', 'runtime': '896160.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896224415 ms)
2025-02-18 04:53:25.704 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+896283523 ms)
2025-02-18 04:53:25.704 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+896283523 ms)
2025-02-18 04:53:25.705 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36270)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896283524 ms)
2025-02-18 04:53:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896284412 ms)
2025-02-18 04:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896284412 ms)
2025-02-18 04:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:53:25.400626+00:00', 'runtime': '896280.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.613', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896284413 ms)
2025-02-18 04:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:52:25.400588+00:00', 'runtime': '896220.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.616', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896284414 ms)
2025-02-18 04:54:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896344415 ms)
2025-02-18 04:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896344415 ms)
2025-02-18 04:54:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:54:25.400170+00:00', 'runtime': '896340.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.604', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896344419 ms)
2025-02-18 04:54:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:53:25.400626+00:00', 'runtime': '896280.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.613', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20135', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896344419 ms)
2025-02-18 04:54:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896353419 ms)
2025-02-18 04:54:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896353419 ms)
2025-02-18 04:54:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:54:25.405064+00:00'} (T+896353424 ms)
2025-02-18 04:55:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896404418 ms)
2025-02-18 04:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896404418 ms)
2025-02-18 04:55:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:55:25.400202+00:00', 'runtime': '896400.001', 'adc_vb_f': '12.616', 'adc_va_f': '1.071', 'adc_vl_f': '12.607', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896404422 ms)
2025-02-18 04:55:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:54:25.400170+00:00', 'runtime': '896340.001', 'adc_vb_f': '12.619', 'adc_va_f': '1.071', 'adc_vl_f': '12.604', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.616', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896404423 ms)
2025-02-18 04:55:35.594 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896413413 ms)
2025-02-18 04:55:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896413414 ms)
2025-02-18 04:55:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:55:25.400221+00:00'} (T+896413419 ms)
2025-02-18 04:56:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896464418 ms)
2025-02-18 04:56:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896464418 ms)
2025-02-18 04:56:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:56:25.400267+00:00', 'runtime': '896460.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896464420 ms)
2025-02-18 04:56:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:55:25.400202+00:00', 'runtime': '896400.001', 'adc_vb_f': '12.616', 'adc_va_f': '1.071', 'adc_vl_f': '12.607', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896464421 ms)
2025-02-18 04:56:35.595 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896473414 ms)
2025-02-18 04:56:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896473415 ms)
2025-02-18 04:56:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:56:25.400276+00:00'} (T+896473416 ms)
2025-02-18 04:57:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896524405 ms)
2025-02-18 04:57:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896524406 ms)
2025-02-18 04:57:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:57:25.400146+00:00', 'runtime': '896520.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.592', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896524408 ms)
2025-02-18 04:57:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:56:25.400267+00:00', 'runtime': '896460.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.3', 'ahl_t': '1595.3', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896524408 ms)
2025-02-18 04:57:35.592 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896533411 ms)
2025-02-18 04:57:35.592 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896533412 ms)
2025-02-18 04:57:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:57:25.400203+00:00'} (T+896533418 ms)
2025-02-18 04:58:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896584412 ms)
2025-02-18 04:58:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896584412 ms)
2025-02-18 04:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:58:25.400288+00:00', 'runtime': '896580.001', 'adc_vb_f': '12.61', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896584414 ms)
2025-02-18 04:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:57:25.400146+00:00', 'runtime': '896520.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.592', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896584415 ms)
2025-02-18 04:58:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896593425 ms)
2025-02-18 04:58:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896593426 ms)
2025-02-18 04:58:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:58:25.400282+00:00'} (T+896593431 ms)
2025-02-18 04:59:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896644414 ms)
2025-02-18 04:59:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896644415 ms)
2025-02-18 04:59:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 04:59:25.400204+00:00', 'runtime': '896640.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896644417 ms)
2025-02-18 04:59:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:58:25.400288+00:00', 'runtime': '896580.001', 'adc_vb_f': '12.61', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896644417 ms)
2025-02-18 04:59:35.638 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896653457 ms)
2025-02-18 04:59:35.639 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896653458 ms)
2025-02-18 04:59:35.643 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 04:59:25.400198+00:00'} (T+896653462 ms)
2025-02-18 05:00:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896704414 ms)
2025-02-18 05:00:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896704414 ms)
2025-02-18 05:00:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:00:25.400186+00:00', 'runtime': '896700.001', 'adc_vb_f': '12.604', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.61', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896704419 ms)
2025-02-18 05:00:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 04:59:25.400204+00:00', 'runtime': '896640.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896704419 ms)
2025-02-18 05:00:35.703 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896713523 ms)
2025-02-18 05:00:35.704 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896713523 ms)
2025-02-18 05:00:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:00:25.400160+00:00'} (T+896713525 ms)
2025-02-18 05:00:38.237 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+896716057 ms)
2025-02-18 05:00:38.238 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+896716057 ms)
2025-02-18 05:00:38.238 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896716057 ms)
2025-02-18 05:00:38.239 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+896716058 ms)
2025-02-18 05:00:38.239 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+896716058 ms)
2025-02-18 05:00:38.239 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896716059 ms)
2025-02-18 05:00:38.240 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+896716059 ms)
2025-02-18 05:00:38.240 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+896716059 ms)
2025-02-18 05:00:38.240 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896716060 ms)
2025-02-18 05:00:38.243 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+896716062 ms)
2025-02-18 05:00:38.243 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+896716062 ms)
2025-02-18 05:00:38.244 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896716063 ms)
2025-02-18 05:00:38.245 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+896716064 ms)
2025-02-18 05:00:38.245 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+896716065 ms)
2025-02-18 05:00:38.246 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896716065 ms)
2025-02-18 05:00:38.247 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+896716066 ms)
2025-02-18 05:00:38.247 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+896716066 ms)
2025-02-18 05:00:38.248 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896716067 ms)
2025-02-18 05:00:38.249 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+896716068 ms)
2025-02-18 05:00:38.251 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+896716070 ms)
2025-02-18 05:00:38.257 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896716077 ms)
2025-02-18 05:01:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896764418 ms)
2025-02-18 05:01:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896764418 ms)
2025-02-18 05:01:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:01:25.402702+00:00', 'runtime': '896760.003', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.61', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896764422 ms)
2025-02-18 05:01:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:00:25.400186+00:00', 'runtime': '896700.001', 'adc_vb_f': '12.604', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.61', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896764423 ms)
2025-02-18 05:01:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896773430 ms)
2025-02-18 05:01:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+896773431 ms)
2025-02-18 05:01:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:01:25.400089+00:00'} (T+896773432 ms)
2025-02-18 05:02:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896824422 ms)
2025-02-18 05:02:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896824423 ms)
2025-02-18 05:02:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:02:25.400241+00:00', 'runtime': '896820.001', 'adc_vb_f': '12.61', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.61', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896824427 ms)
2025-02-18 05:02:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:01:25.402702+00:00', 'runtime': '896760.003', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.61', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896824427 ms)
2025-02-18 05:03:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896884411 ms)
2025-02-18 05:03:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896884411 ms)
2025-02-18 05:03:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:03:25.400142+00:00', 'runtime': '896880.001', 'adc_vb_f': '12.616', 'adc_va_f': '1.071', 'adc_vl_f': '12.604', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.607', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896884412 ms)
2025-02-18 05:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:02:25.400241+00:00', 'runtime': '896820.001', 'adc_vb_f': '12.61', 'adc_va_f': '1.071', 'adc_vl_f': '12.589', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.61', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896884415 ms)
2025-02-18 05:04:00.586 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+896918406 ms)
2025-02-18 05:04:00.587 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+896918406 ms)
2025-02-18 05:04:00.587 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36956)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+896918406 ms)
2025-02-18 05:04:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+896944416 ms)
2025-02-18 05:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+896944417 ms)
2025-02-18 05:04:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:04:25.400172+00:00', 'runtime': '896940.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.604', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896944420 ms)
2025-02-18 05:04:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:03:25.400142+00:00', 'runtime': '896880.001', 'adc_vb_f': '12.616', 'adc_va_f': '1.071', 'adc_vl_f': '12.604', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.607', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+896944420 ms)
2025-02-18 05:05:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897004420 ms)
2025-02-18 05:05:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897004420 ms)
2025-02-18 05:05:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:05:25.400525+00:00', 'runtime': '897000.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897004424 ms)
2025-02-18 05:05:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:04:25.400172+00:00', 'runtime': '896940.001', 'adc_vb_f': '12.607', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.604', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897004425 ms)
2025-02-18 05:05:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897013432 ms)
2025-02-18 05:05:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897013433 ms)
2025-02-18 05:05:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:05:25.409863+00:00'} (T+897013434 ms)
2025-02-18 05:06:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897064406 ms)
2025-02-18 05:06:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897064407 ms)
2025-02-18 05:06:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:06:25.400159+00:00', 'runtime': '897060.001', 'adc_vb_f': '12.604', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897064409 ms)
2025-02-18 05:06:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:05:25.400525+00:00', 'runtime': '897000.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897064409 ms)
2025-02-18 05:06:36.237 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897074056 ms)
2025-02-18 05:06:36.238 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897074057 ms)
2025-02-18 05:06:36.239 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:06:25.409116+00:00'} (T+897074058 ms)
2025-02-18 05:07:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897124411 ms)
2025-02-18 05:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897124411 ms)
2025-02-18 05:07:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:07:25.402666+00:00', 'runtime': '897120.003', 'adc_vb_f': '12.604', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897124416 ms)
2025-02-18 05:07:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:06:25.400159+00:00', 'runtime': '897060.001', 'adc_vb_f': '12.604', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897124416 ms)
2025-02-18 05:07:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897133417 ms)
2025-02-18 05:07:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897133418 ms)
2025-02-18 05:07:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:07:25.400050+00:00'} (T+897133419 ms)
2025-02-18 05:07:59.969 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+897157788 ms)
2025-02-18 05:07:59.969 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+897157788 ms)
2025-02-18 05:07:59.970 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37220)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897157789 ms)
2025-02-18 05:08:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897184416 ms)
2025-02-18 05:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897184417 ms)
2025-02-18 05:08:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:08:25.400252+00:00', 'runtime': '897180.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897184420 ms)
2025-02-18 05:08:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:07:25.402666+00:00', 'runtime': '897120.003', 'adc_vb_f': '12.604', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897184420 ms)
2025-02-18 05:08:35.635 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897193454 ms)
2025-02-18 05:08:35.635 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897193455 ms)
2025-02-18 05:08:35.636 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:08:25.413122+00:00'} (T+897193456 ms)
2025-02-18 05:09:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897244419 ms)
2025-02-18 05:09:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897244420 ms)
2025-02-18 05:09:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:09:25.402783+00:00', 'runtime': '897240.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897244424 ms)
2025-02-18 05:09:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:08:25.400252+00:00', 'runtime': '897180.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897244425 ms)
2025-02-18 05:09:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897253433 ms)
2025-02-18 05:09:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897253433 ms)
2025-02-18 05:09:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:09:25.400168+00:00'} (T+897253434 ms)
2025-02-18 05:09:57.208 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+897275028 ms)
2025-02-18 05:09:57.209 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+897275028 ms)
2025-02-18 05:09:57.209 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37346)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897275028 ms)
2025-02-18 05:10:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897304422 ms)
2025-02-18 05:10:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897304422 ms)
2025-02-18 05:10:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:10:25.400172+00:00', 'runtime': '897300.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897304425 ms)
2025-02-18 05:10:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:09:25.402783+00:00', 'runtime': '897240.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897304426 ms)
2025-02-18 05:10:35.683 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897313502 ms)
2025-02-18 05:10:35.683 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897313503 ms)
2025-02-18 05:10:35.684 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:10:25.413127+00:00'} (T+897313504 ms)
2025-02-18 05:11:15.310 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897353129 ms)
2025-02-18 05:11:15.310 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897353129 ms)
2025-02-18 05:11:15.310 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897353129 ms)
2025-02-18 05:11:15.311 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897353130 ms)
2025-02-18 05:11:15.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897353130 ms)
2025-02-18 05:11:15.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897353131 ms)
2025-02-18 05:11:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897364408 ms)
2025-02-18 05:11:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897364408 ms)
2025-02-18 05:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:11:25.402768+00:00', 'runtime': '897360.003', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897364411 ms)
2025-02-18 05:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:10:25.400172+00:00', 'runtime': '897300.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.4', 'ahl_t': '1595.4', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897364411 ms)
2025-02-18 05:11:35.783 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897373602 ms)
2025-02-18 05:11:35.783 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897373603 ms)
2025-02-18 05:11:35.784 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:11:25.400126+00:00'} (T+897373604 ms)
2025-02-18 05:12:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897424411 ms)
2025-02-18 05:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897424411 ms)
2025-02-18 05:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:12:25.402610+00:00', 'runtime': '897420.003', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897424415 ms)
2025-02-18 05:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:11:25.402768+00:00', 'runtime': '897360.003', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897424416 ms)
2025-02-18 05:12:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897433421 ms)
2025-02-18 05:12:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897433422 ms)
2025-02-18 05:12:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:12:25.400026+00:00'} (T+897433423 ms)
2025-02-18 05:13:15.461 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897473281 ms)
2025-02-18 05:13:15.462 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897473281 ms)
2025-02-18 05:13:15.462 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897473281 ms)
2025-02-18 05:13:15.463 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897473282 ms)
2025-02-18 05:13:15.463 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897473282 ms)
2025-02-18 05:13:15.463 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897473283 ms)
2025-02-18 05:13:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897484413 ms)
2025-02-18 05:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897484414 ms)
2025-02-18 05:13:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:13:25.400164+00:00', 'runtime': '897480.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897484417 ms)
2025-02-18 05:13:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:12:25.402610+00:00', 'runtime': '897420.003', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897484418 ms)
2025-02-18 05:13:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897493427 ms)
2025-02-18 05:13:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897493428 ms)
2025-02-18 05:13:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:13:25.405134+00:00'} (T+897493432 ms)
2025-02-18 05:13:56.607 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+897514427 ms)
2025-02-18 05:13:56.608 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+897514427 ms)
2025-02-18 05:13:56.608 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37602)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897514427 ms)
2025-02-18 05:14:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897544423 ms)
2025-02-18 05:14:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897544424 ms)
2025-02-18 05:14:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:14:25.400165+00:00', 'runtime': '897540.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897544427 ms)
2025-02-18 05:14:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:13:25.400164+00:00', 'runtime': '897480.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897544427 ms)
2025-02-18 05:14:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897553431 ms)
2025-02-18 05:14:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897553432 ms)
2025-02-18 05:14:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:14:25.413124+00:00'} (T+897553433 ms)
2025-02-18 05:15:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897604421 ms)
2025-02-18 05:15:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897604421 ms)
2025-02-18 05:15:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:15:25.400165+00:00', 'runtime': '897600.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897604426 ms)
2025-02-18 05:15:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:14:25.400165+00:00', 'runtime': '897540.001', 'adc_vb_f': '12.601', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897604426 ms)
2025-02-18 05:15:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897613433 ms)
2025-02-18 05:15:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897613434 ms)
2025-02-18 05:15:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:15:25.405134+00:00'} (T+897613435 ms)
2025-02-18 05:16:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897664410 ms)
2025-02-18 05:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897664410 ms)
2025-02-18 05:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:16:25.400170+00:00', 'runtime': '897660.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897664412 ms)
2025-02-18 05:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:15:25.400165+00:00', 'runtime': '897600.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897664413 ms)
2025-02-18 05:16:35.596 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897673415 ms)
2025-02-18 05:16:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897673415 ms)
2025-02-18 05:16:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:16:25.413125+00:00'} (T+897673416 ms)
2025-02-18 05:17:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897724415 ms)
2025-02-18 05:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897724416 ms)
2025-02-18 05:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:17:25.400166+00:00', 'runtime': '897720.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897724418 ms)
2025-02-18 05:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:16:25.400170+00:00', 'runtime': '897660.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897724418 ms)
2025-02-18 05:17:35.593 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897733412 ms)
2025-02-18 05:17:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+897733413 ms)
2025-02-18 05:17:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:17:25.405135+00:00'} (T+897733414 ms)
2025-02-18 05:18:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897784417 ms)
2025-02-18 05:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897784417 ms)
2025-02-18 05:18:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:18:25.400163+00:00', 'runtime': '897780.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897784422 ms)
2025-02-18 05:18:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:17:25.400166+00:00', 'runtime': '897720.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897784422 ms)
2025-02-18 05:18:50.974 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897808793 ms)
2025-02-18 05:18:50.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897808793 ms)
2025-02-18 05:18:50.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897808794 ms)
2025-02-18 05:18:50.975 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897808794 ms)
2025-02-18 05:18:50.975 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897808795 ms)
2025-02-18 05:18:50.976 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897808795 ms)
2025-02-18 05:18:50.977 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897808796 ms)
2025-02-18 05:18:50.977 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897808797 ms)
2025-02-18 05:18:50.979 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897808799 ms)
2025-02-18 05:18:50.982 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+897808801 ms)
2025-02-18 05:18:50.982 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+897808802 ms)
2025-02-18 05:18:50.988 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897808807 ms)
2025-02-18 05:19:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897844401 ms)
2025-02-18 05:19:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897844402 ms)
2025-02-18 05:19:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:19:25.400934+00:00', 'runtime': '897840.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897844403 ms)
2025-02-18 05:19:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:18:25.400163+00:00', 'runtime': '897780.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897844406 ms)
2025-02-18 05:19:47.145 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+897864964 ms)
2025-02-18 05:19:47.145 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+897864965 ms)
2025-02-18 05:19:47.146 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37984)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897864965 ms)
2025-02-18 05:20:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897904406 ms)
2025-02-18 05:20:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897904407 ms)
2025-02-18 05:20:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:20:25.400226+00:00', 'runtime': '897900.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897904408 ms)
2025-02-18 05:20:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:19:25.400934+00:00', 'runtime': '897840.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897904408 ms)
2025-02-18 05:20:46.320 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+897924139 ms)
2025-02-18 05:20:46.320 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+897924139 ms)
2025-02-18 05:20:46.320 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38046)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897924140 ms)
2025-02-18 05:20:59.423 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+897937242 ms)
2025-02-18 05:20:59.423 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+897937242 ms)
2025-02-18 05:20:59.423 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38062)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+897937243 ms)
2025-02-18 05:21:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+897964412 ms)
2025-02-18 05:21:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+897964412 ms)
2025-02-18 05:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:21:25.400181+00:00', 'runtime': '897960.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897964414 ms)
2025-02-18 05:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:20:25.400226+00:00', 'runtime': '897900.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+897964415 ms)
2025-02-18 05:22:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898024413 ms)
2025-02-18 05:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898024414 ms)
2025-02-18 05:22:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:22:25.400178+00:00', 'runtime': '898020.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898024415 ms)
2025-02-18 05:22:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:21:25.400181+00:00', 'runtime': '897960.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898024416 ms)
2025-02-18 05:23:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898084417 ms)
2025-02-18 05:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898084418 ms)
2025-02-18 05:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:23:25.400589+00:00', 'runtime': '898080.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898084419 ms)
2025-02-18 05:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:22:25.400178+00:00', 'runtime': '898020.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898084420 ms)
2025-02-18 05:24:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898144422 ms)
2025-02-18 05:24:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898144423 ms)
2025-02-18 05:24:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:24:25.402799+00:00', 'runtime': '898140.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898144425 ms)
2025-02-18 05:24:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:23:25.400589+00:00', 'runtime': '898080.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898144425 ms)
2025-02-18 05:24:53.762 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+898171581 ms)
2025-02-18 05:24:53.763 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+898171582 ms)
2025-02-18 05:24:53.763 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38310)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+898171582 ms)
2025-02-18 05:25:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898204411 ms)
2025-02-18 05:25:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898204411 ms)
2025-02-18 05:25:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:25:25.400170+00:00', 'runtime': '898200.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898204413 ms)
2025-02-18 05:25:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:24:25.402799+00:00', 'runtime': '898140.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.5', 'ahl_t': '1595.5', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898204414 ms)
2025-02-18 05:26:11.924 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+898249743 ms)
2025-02-18 05:26:11.924 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+898249743 ms)
2025-02-18 05:26:11.924 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38402)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+898249744 ms)
2025-02-18 05:26:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898264414 ms)
2025-02-18 05:26:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898264414 ms)
2025-02-18 05:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:26:25.400623+00:00', 'runtime': '898260.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898264416 ms)
2025-02-18 05:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:25:25.400170+00:00', 'runtime': '898200.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898264416 ms)
2025-02-18 05:27:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898324417 ms)
2025-02-18 05:27:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898324418 ms)
2025-02-18 05:27:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:27:25.402799+00:00', 'runtime': '898320.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898324419 ms)
2025-02-18 05:27:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:26:25.400623+00:00', 'runtime': '898260.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898324420 ms)
2025-02-18 05:28:24.155 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+898381974 ms)
2025-02-18 05:28:24.155 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+898381975 ms)
2025-02-18 05:28:24.156 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38542)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+898381975 ms)
2025-02-18 05:28:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898384408 ms)
2025-02-18 05:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898384409 ms)
2025-02-18 05:28:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:28:25.400167+00:00', 'runtime': '898380.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898384411 ms)
2025-02-18 05:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:27:25.402799+00:00', 'runtime': '898320.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898384411 ms)
2025-02-18 05:28:39.266 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+898397085 ms)
2025-02-18 05:28:39.266 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+898397086 ms)
2025-02-18 05:28:39.267 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38558)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+898397086 ms)
2025-02-18 05:29:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898444414 ms)
2025-02-18 05:29:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898444414 ms)
2025-02-18 05:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:29:25.400226+00:00', 'runtime': '898440.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898444416 ms)
2025-02-18 05:29:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:28:25.400167+00:00', 'runtime': '898380.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.071', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898444417 ms)
2025-02-18 05:29:35.697 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898453517 ms)
2025-02-18 05:29:35.698 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898453517 ms)
2025-02-18 05:29:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:29:25.401146+00:00'} (T+898453518 ms)
2025-02-18 05:30:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898504417 ms)
2025-02-18 05:30:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898504418 ms)
2025-02-18 05:30:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:30:25.401138+00:00', 'runtime': '898500.002', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898504422 ms)
2025-02-18 05:30:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:29:25.400226+00:00', 'runtime': '898440.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898504423 ms)
2025-02-18 05:30:35.704 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898513524 ms)
2025-02-18 05:30:35.705 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898513524 ms)
2025-02-18 05:30:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:30:25.400116+00:00'} (T+898513526 ms)
2025-02-18 05:31:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898564406 ms)
2025-02-18 05:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898564406 ms)
2025-02-18 05:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:31:25.400185+00:00', 'runtime': '898560.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.077', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898564407 ms)
2025-02-18 05:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:30:25.401138+00:00', 'runtime': '898500.002', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898564408 ms)
2025-02-18 05:31:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898573423 ms)
2025-02-18 05:31:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898573423 ms)
2025-02-18 05:31:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:31:25.400116+00:00'} (T+898573424 ms)
2025-02-18 05:32:26.607 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898624426 ms)
2025-02-18 05:32:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898624426 ms)
2025-02-18 05:32:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:32:25.400199+00:00', 'runtime': '898620.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898624427 ms)
2025-02-18 05:32:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:31:25.400185+00:00', 'runtime': '898560.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.077', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898624428 ms)
2025-02-18 05:32:35.626 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898633445 ms)
2025-02-18 05:32:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898633445 ms)
2025-02-18 05:32:35.627 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:32:25.400575+00:00'} (T+898633446 ms)
2025-02-18 05:33:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898684414 ms)
2025-02-18 05:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898684414 ms)
2025-02-18 05:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:33:25.400171+00:00', 'runtime': '898680.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.083', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898684415 ms)
2025-02-18 05:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:32:25.400199+00:00', 'runtime': '898620.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.074', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898684416 ms)
2025-02-18 05:33:35.691 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898693510 ms)
2025-02-18 05:33:35.691 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+898693511 ms)
2025-02-18 05:33:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:33:25.400183+00:00'} (T+898693512 ms)
2025-02-18 05:34:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898744421 ms)
2025-02-18 05:34:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898744421 ms)
2025-02-18 05:34:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:34:25.402735+00:00', 'runtime': '898740.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898744425 ms)
2025-02-18 05:34:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:33:25.400171+00:00', 'runtime': '898680.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.083', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898744426 ms)
2025-02-18 05:35:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898804423 ms)
2025-02-18 05:35:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898804423 ms)
2025-02-18 05:35:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:35:25.400158+00:00', 'runtime': '898800.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.089', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898804427 ms)
2025-02-18 05:35:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:34:25.402735+00:00', 'runtime': '898740.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898804427 ms)
2025-02-18 05:36:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898864413 ms)
2025-02-18 05:36:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898864413 ms)
2025-02-18 05:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:36:25.402742+00:00', 'runtime': '898860.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898864416 ms)
2025-02-18 05:36:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:35:25.400158+00:00', 'runtime': '898800.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.089', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898864417 ms)
2025-02-18 05:37:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898924414 ms)
2025-02-18 05:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898924415 ms)
2025-02-18 05:37:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:37:25.400178+00:00', 'runtime': '898920.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898924416 ms)
2025-02-18 05:37:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:36:25.402742+00:00', 'runtime': '898860.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898924418 ms)
2025-02-18 05:38:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+898984418 ms)
2025-02-18 05:38:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+898984419 ms)
2025-02-18 05:38:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:38:25.400155+00:00', 'runtime': '898980.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.002', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898984420 ms)
2025-02-18 05:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:37:25.400178+00:00', 'runtime': '898920.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+898984423 ms)
2025-02-18 05:39:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899044404 ms)
2025-02-18 05:39:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899044404 ms)
2025-02-18 05:39:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:39:25.400167+00:00', 'runtime': '899040.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.089', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899044408 ms)
2025-02-18 05:39:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:38:25.400155+00:00', 'runtime': '898980.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.002', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.6', 'ahl_t': '1595.6', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899044409 ms)
2025-02-18 05:40:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899104410 ms)
2025-02-18 05:40:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899104410 ms)
2025-02-18 05:40:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:40:25.400205+00:00', 'runtime': '899100.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899104415 ms)
2025-02-18 05:40:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:39:25.400167+00:00', 'runtime': '899040.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.089', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899104415 ms)
2025-02-18 05:41:04.247 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+899142067 ms)
2025-02-18 05:41:04.248 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+899142067 ms)
2025-02-18 05:41:04.248 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39358)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899142067 ms)
2025-02-18 05:41:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899164414 ms)
2025-02-18 05:41:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899164414 ms)
2025-02-18 05:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:41:25.400201+00:00', 'runtime': '899160.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.096', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899164418 ms)
2025-02-18 05:41:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:40:25.400205+00:00', 'runtime': '899100.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899164418 ms)
2025-02-18 05:41:46.377 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+899184196 ms)
2025-02-18 05:41:46.377 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+899184196 ms)
2025-02-18 05:41:46.377 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39402)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899184197 ms)
2025-02-18 05:42:16.416 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899214235 ms)
2025-02-18 05:42:16.416 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899214236 ms)
2025-02-18 05:42:16.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899214236 ms)
2025-02-18 05:42:16.417 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899214237 ms)
2025-02-18 05:42:16.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899214237 ms)
2025-02-18 05:42:16.418 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899214237 ms)
2025-02-18 05:42:16.418 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899214237 ms)
2025-02-18 05:42:16.418 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899214238 ms)
2025-02-18 05:42:16.419 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899214238 ms)
2025-02-18 05:42:16.419 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899214238 ms)
2025-02-18 05:42:16.419 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899214238 ms)
2025-02-18 05:42:16.419 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899214239 ms)
2025-02-18 05:42:16.420 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899214239 ms)
2025-02-18 05:42:16.420 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899214239 ms)
2025-02-18 05:42:16.420 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899214239 ms)
2025-02-18 05:42:16.420 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899214240 ms)
2025-02-18 05:42:16.421 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899214240 ms)
2025-02-18 05:42:16.421 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899214240 ms)
2025-02-18 05:42:16.423 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899214243 ms)
2025-02-18 05:42:16.424 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899214243 ms)
2025-02-18 05:42:16.424 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899214244 ms)
2025-02-18 05:42:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899224416 ms)
2025-02-18 05:42:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899224417 ms)
2025-02-18 05:42:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:42:25.400171+00:00', 'runtime': '899220.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899224420 ms)
2025-02-18 05:42:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:41:25.400201+00:00', 'runtime': '899160.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.096', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899224420 ms)
2025-02-18 05:42:35.700 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+899233520 ms)
2025-02-18 05:42:35.701 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+899233520 ms)
2025-02-18 05:42:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:42:25.400346+00:00'} (T+899233525 ms)
2025-02-18 05:43:15.576 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+899273395 ms)
2025-02-18 05:43:15.576 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+899273396 ms)
2025-02-18 05:43:15.577 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39498)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899273396 ms)
2025-02-18 05:43:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899284405 ms)
2025-02-18 05:43:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899284406 ms)
2025-02-18 05:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:43:25.400165+00:00', 'runtime': '899280.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.102', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899284409 ms)
2025-02-18 05:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:42:25.400171+00:00', 'runtime': '899220.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.093', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899284409 ms)
2025-02-18 05:43:35.586 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+899293405 ms)
2025-02-18 05:43:35.586 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+899293406 ms)
2025-02-18 05:43:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:43:25.400100+00:00'} (T+899293410 ms)
2025-02-18 05:44:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899344409 ms)
2025-02-18 05:44:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899344409 ms)
2025-02-18 05:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:44:25.400225+00:00', 'runtime': '899340.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.12', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899344411 ms)
2025-02-18 05:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:43:25.400165+00:00', 'runtime': '899280.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.102', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899344412 ms)
2025-02-18 05:45:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899404413 ms)
2025-02-18 05:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899404413 ms)
2025-02-18 05:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:45:25.400173+00:00', 'runtime': '899400.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.096', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899404415 ms)
2025-02-18 05:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:44:25.400225+00:00', 'runtime': '899340.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.12', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899404415 ms)
2025-02-18 05:46:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899464417 ms)
2025-02-18 05:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899464418 ms)
2025-02-18 05:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:46:25.400174+00:00', 'runtime': '899460.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.114', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899464420 ms)
2025-02-18 05:46:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:45:25.400173+00:00', 'runtime': '899400.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.096', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899464420 ms)
2025-02-18 05:47:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899524406 ms)
2025-02-18 05:47:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899524406 ms)
2025-02-18 05:47:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:47:25.400248+00:00', 'runtime': '899520.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.114', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899524408 ms)
2025-02-18 05:47:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:46:25.400174+00:00', 'runtime': '899460.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.114', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899524409 ms)
2025-02-18 05:48:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899584409 ms)
2025-02-18 05:48:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899584410 ms)
2025-02-18 05:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:48:25.400190+00:00', 'runtime': '899580.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899584411 ms)
2025-02-18 05:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:47:25.400248+00:00', 'runtime': '899520.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.114', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899584412 ms)
2025-02-18 05:49:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899644414 ms)
2025-02-18 05:49:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899644414 ms)
2025-02-18 05:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:49:25.400151+00:00', 'runtime': '899640.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899644415 ms)
2025-02-18 05:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:48:25.400190+00:00', 'runtime': '899580.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899644416 ms)
2025-02-18 05:50:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899704417 ms)
2025-02-18 05:50:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899704417 ms)
2025-02-18 05:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:50:25.400521+00:00', 'runtime': '899700.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899704419 ms)
2025-02-18 05:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:49:25.400151+00:00', 'runtime': '899640.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899704420 ms)
2025-02-18 05:51:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899764406 ms)
2025-02-18 05:51:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899764406 ms)
2025-02-18 05:51:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:51:25.400223+00:00', 'runtime': '899760.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.114', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899764408 ms)
2025-02-18 05:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:50:25.400521+00:00', 'runtime': '899700.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899764409 ms)
2025-02-18 05:52:02.221 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899800041 ms)
2025-02-18 05:52:02.222 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899800041 ms)
2025-02-18 05:52:02.222 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899800041 ms)
2025-02-18 05:52:02.223 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+899800042 ms)
2025-02-18 05:52:02.223 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+899800042 ms)
2025-02-18 05:52:02.223 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899800043 ms)
2025-02-18 05:52:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899824409 ms)
2025-02-18 05:52:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899824409 ms)
2025-02-18 05:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:52:25.400199+00:00', 'runtime': '899820.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899824411 ms)
2025-02-18 05:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:51:25.400223+00:00', 'runtime': '899760.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.114', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.7', 'ahl_t': '1595.7', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899824412 ms)
2025-02-18 05:52:44.348 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+899842167 ms)
2025-02-18 05:52:44.348 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+899842168 ms)
2025-02-18 05:52:44.349 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40106)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+899842168 ms)
2025-02-18 05:53:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899884414 ms)
2025-02-18 05:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899884414 ms)
2025-02-18 05:53:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:53:25.400213+00:00', 'runtime': '899880.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899884416 ms)
2025-02-18 05:53:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:52:25.400199+00:00', 'runtime': '899820.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899884417 ms)
2025-02-18 05:54:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+899944417 ms)
2025-02-18 05:54:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+899944417 ms)
2025-02-18 05:54:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:54:25.400175+00:00', 'runtime': '899940.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899944419 ms)
2025-02-18 05:54:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:53:25.400213+00:00', 'runtime': '899880.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20136', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+899944420 ms)
2025-02-18 05:55:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900004419 ms)
2025-02-18 05:55:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900004420 ms)
2025-02-18 05:55:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:55:25.400810+00:00', 'runtime': '900000.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900004422 ms)
2025-02-18 05:55:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:54:25.400175+00:00', 'runtime': '899940.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900004423 ms)
2025-02-18 05:55:27.618 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+900005437 ms)
2025-02-18 05:55:27.618 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+900005437 ms)
2025-02-18 05:55:27.619 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40278)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+900005438 ms)
2025-02-18 05:56:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900064406 ms)
2025-02-18 05:56:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900064406 ms)
2025-02-18 05:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:56:25.401002+00:00', 'runtime': '900060.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900064408 ms)
2025-02-18 05:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:55:25.400810+00:00', 'runtime': '900000.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900064409 ms)
2025-02-18 05:57:04.835 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+900102654 ms)
2025-02-18 05:57:04.835 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+900102654 ms)
2025-02-18 05:57:04.835 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40390)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+900102655 ms)
2025-02-18 05:57:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900124410 ms)
2025-02-18 05:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900124410 ms)
2025-02-18 05:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:57:25.400208+00:00', 'runtime': '900120.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900124412 ms)
2025-02-18 05:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:56:25.401002+00:00', 'runtime': '900060.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900124413 ms)
2025-02-18 05:57:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900133420 ms)
2025-02-18 05:57:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900133421 ms)
2025-02-18 05:57:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 05:57:25.413050+00:00'} (T+900133423 ms)
2025-02-18 05:57:52.969 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+900150789 ms)
2025-02-18 05:57:52.970 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+900150789 ms)
2025-02-18 05:57:52.970 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40444)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+900150789 ms)
2025-02-18 05:58:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900184412 ms)
2025-02-18 05:58:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900184412 ms)
2025-02-18 05:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:58:25.400508+00:00', 'runtime': '900180.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.12', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900184414 ms)
2025-02-18 05:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:57:25.400208+00:00', 'runtime': '900120.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900184415 ms)
2025-02-18 05:59:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900244415 ms)
2025-02-18 05:59:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900244416 ms)
2025-02-18 05:59:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 05:59:25.400178+00:00', 'runtime': '900240.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900244418 ms)
2025-02-18 05:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:58:25.400508+00:00', 'runtime': '900180.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.12', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900244418 ms)
2025-02-18 06:00:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900304418 ms)
2025-02-18 06:00:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900304419 ms)
2025-02-18 06:00:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:00:25.400179+00:00', 'runtime': '900300.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.126', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900304420 ms)
2025-02-18 06:00:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 05:59:25.400178+00:00', 'runtime': '900240.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900304420 ms)
2025-02-18 06:00:35.589 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900313409 ms)
2025-02-18 06:00:35.590 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900313409 ms)
2025-02-18 06:00:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:00:25.405069+00:00'} (T+900313411 ms)
2025-02-18 06:00:45.264 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+900323083 ms)
2025-02-18 06:00:45.264 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+900323084 ms)
2025-02-18 06:00:45.265 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40626)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+900323084 ms)
2025-02-18 06:01:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900364406 ms)
2025-02-18 06:01:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900364406 ms)
2025-02-18 06:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:01:25.400172+00:00', 'runtime': '900360.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.123', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900364407 ms)
2025-02-18 06:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:00:25.400179+00:00', 'runtime': '900300.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.126', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900364408 ms)
2025-02-18 06:01:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900373419 ms)
2025-02-18 06:01:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900373420 ms)
2025-02-18 06:01:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:01:25.405213+00:00'} (T+900373421 ms)
2025-02-18 06:02:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900424408 ms)
2025-02-18 06:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900424408 ms)
2025-02-18 06:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:02:25.402872+00:00', 'runtime': '900420.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.123', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900424412 ms)
2025-02-18 06:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:01:25.400172+00:00', 'runtime': '900360.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.123', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900424413 ms)
2025-02-18 06:02:33.423 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+900431242 ms)
2025-02-18 06:03:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900484411 ms)
2025-02-18 06:03:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900484412 ms)
2025-02-18 06:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:03:25.400781+00:00', 'runtime': '900480.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.117', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900484416 ms)
2025-02-18 06:03:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:02:25.402872+00:00', 'runtime': '900420.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.123', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900484416 ms)
2025-02-18 06:04:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900544413 ms)
2025-02-18 06:04:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900544413 ms)
2025-02-18 06:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:04:25.400176+00:00', 'runtime': '900540.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900544416 ms)
2025-02-18 06:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:03:25.400781+00:00', 'runtime': '900480.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.117', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900544416 ms)
2025-02-18 06:04:35.688 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900553507 ms)
2025-02-18 06:04:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900553508 ms)
2025-02-18 06:04:35.691 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:04:25.401086+00:00'} (T+900553510 ms)
2025-02-18 06:05:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900604418 ms)
2025-02-18 06:05:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900604419 ms)
2025-02-18 06:05:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:05:25.401207+00:00', 'runtime': '900600.002', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900604420 ms)
2025-02-18 06:05:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:04:25.400176+00:00', 'runtime': '900540.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900604421 ms)
2025-02-18 06:05:36.328 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900614148 ms)
2025-02-18 06:05:36.329 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900614148 ms)
2025-02-18 06:05:36.330 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:05:25.400105+00:00'} (T+900614149 ms)
2025-02-18 06:06:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900664421 ms)
2025-02-18 06:06:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900664421 ms)
2025-02-18 06:06:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:06:25.400047+00:00', 'runtime': '900660.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900664423 ms)
2025-02-18 06:06:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:05:25.401207+00:00', 'runtime': '900600.002', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.8', 'ahl_t': '1595.8', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.598', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900664424 ms)
2025-02-18 06:06:35.715 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900673534 ms)
2025-02-18 06:06:35.715 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900673535 ms)
2025-02-18 06:06:35.716 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:06:25.409119+00:00'} (T+900673536 ms)
2025-02-18 06:07:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900724408 ms)
2025-02-18 06:07:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900724409 ms)
2025-02-18 06:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:07:25.402711+00:00', 'runtime': '900720.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900724413 ms)
2025-02-18 06:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:06:25.400047+00:00', 'runtime': '900660.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.129', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900724414 ms)
2025-02-18 06:07:35.709 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900733528 ms)
2025-02-18 06:07:35.710 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900733529 ms)
2025-02-18 06:07:35.711 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:07:25.400073+00:00'} (T+900733530 ms)
2025-02-18 06:08:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900784413 ms)
2025-02-18 06:08:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900784413 ms)
2025-02-18 06:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:08:25.402776+00:00', 'runtime': '900780.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900784417 ms)
2025-02-18 06:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:07:25.402711+00:00', 'runtime': '900720.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900784418 ms)
2025-02-18 06:08:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900793428 ms)
2025-02-18 06:08:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900793429 ms)
2025-02-18 06:08:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:08:25.400167+00:00'} (T+900793430 ms)
2025-02-18 06:09:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900844414 ms)
2025-02-18 06:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900844415 ms)
2025-02-18 06:09:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:09:25.400176+00:00', 'runtime': '900840.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900844419 ms)
2025-02-18 06:09:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:08:25.402776+00:00', 'runtime': '900780.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900844419 ms)
2025-02-18 06:09:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900853420 ms)
2025-02-18 06:09:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900853424 ms)
2025-02-18 06:09:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:09:25.413125+00:00'} (T+900853425 ms)
2025-02-18 06:10:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900904417 ms)
2025-02-18 06:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900904418 ms)
2025-02-18 06:10:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:10:25.400157+00:00', 'runtime': '900900.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900904422 ms)
2025-02-18 06:10:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:09:25.400176+00:00', 'runtime': '900840.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900904422 ms)
2025-02-18 06:10:35.717 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900913536 ms)
2025-02-18 06:10:35.720 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900913539 ms)
2025-02-18 06:10:35.721 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:10:25.413126+00:00'} (T+900913540 ms)
2025-02-18 06:11:24.143 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+900961963 ms)
2025-02-18 06:11:24.144 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+900961963 ms)
2025-02-18 06:11:24.144 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41314)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+900961963 ms)
2025-02-18 06:11:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+900964421 ms)
2025-02-18 06:11:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+900964422 ms)
2025-02-18 06:11:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:11:25.402683+00:00', 'runtime': '900960.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900964425 ms)
2025-02-18 06:11:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:10:25.400157+00:00', 'runtime': '900900.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+900964425 ms)
2025-02-18 06:11:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900973428 ms)
2025-02-18 06:11:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+900973429 ms)
2025-02-18 06:11:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:11:25.400064+00:00'} (T+900973430 ms)
2025-02-18 06:12:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901024422 ms)
2025-02-18 06:12:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901024423 ms)
2025-02-18 06:12:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:12:25.400167+00:00', 'runtime': '901020.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901024427 ms)
2025-02-18 06:12:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:11:25.402683+00:00', 'runtime': '900960.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901024427 ms)
2025-02-18 06:12:40.288 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901038107 ms)
2025-02-18 06:12:40.288 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901038107 ms)
2025-02-18 06:12:40.289 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41398)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901038108 ms)
2025-02-18 06:13:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901084410 ms)
2025-02-18 06:13:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901084410 ms)
2025-02-18 06:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:13:25.400055+00:00', 'runtime': '901080.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901084414 ms)
2025-02-18 06:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:12:25.400167+00:00', 'runtime': '901020.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901084415 ms)
2025-02-18 06:13:48.412 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901106231 ms)
2025-02-18 06:13:48.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901106231 ms)
2025-02-18 06:13:48.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41472)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901106231 ms)
2025-02-18 06:14:13.542 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901131361 ms)
2025-02-18 06:14:13.542 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901131362 ms)
2025-02-18 06:14:13.543 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41498)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901131362 ms)
2025-02-18 06:14:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901144414 ms)
2025-02-18 06:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901144414 ms)
2025-02-18 06:14:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:14:25.400199+00:00', 'runtime': '901140.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901144418 ms)
2025-02-18 06:14:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:13:25.400055+00:00', 'runtime': '901080.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901144418 ms)
2025-02-18 06:14:59.672 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901177492 ms)
2025-02-18 06:14:59.673 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901177492 ms)
2025-02-18 06:14:59.673 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41548)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901177492 ms)
2025-02-18 06:15:25.696 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901203515 ms)
2025-02-18 06:15:25.696 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901203516 ms)
2025-02-18 06:15:25.700 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:15:25.400183+00:00', 'runtime': '901200.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901203519 ms)
2025-02-18 06:15:25.700 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:14:25.400199+00:00', 'runtime': '901140.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901203519 ms)
2025-02-18 06:15:48.803 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901226623 ms)
2025-02-18 06:15:48.803 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901226623 ms)
2025-02-18 06:15:48.804 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41610)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901226623 ms)
2025-02-18 06:16:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901264404 ms)
2025-02-18 06:16:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901264405 ms)
2025-02-18 06:16:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:16:25.400633+00:00', 'runtime': '901260.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.585', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901264409 ms)
2025-02-18 06:16:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:15:25.400183+00:00', 'runtime': '901200.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.589', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901264409 ms)
2025-02-18 06:16:29.952 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901267772 ms)
2025-02-18 06:16:29.953 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901267772 ms)
2025-02-18 06:16:29.953 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41654)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901267773 ms)
2025-02-18 06:17:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901324408 ms)
2025-02-18 06:17:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901324408 ms)
2025-02-18 06:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:17:25.400592+00:00', 'runtime': '901320.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.585', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901324412 ms)
2025-02-18 06:17:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:16:25.400633+00:00', 'runtime': '901260.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.585', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901324412 ms)
2025-02-18 06:18:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901384412 ms)
2025-02-18 06:18:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901384412 ms)
2025-02-18 06:18:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:18:25.402782+00:00', 'runtime': '901380.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.585', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901384417 ms)
2025-02-18 06:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:17:25.400592+00:00', 'runtime': '901320.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.132', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.585', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901384417 ms)
2025-02-18 06:18:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+901393429 ms)
2025-02-18 06:18:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+901393429 ms)
2025-02-18 06:18:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:18:25.400148+00:00'} (T+901393430 ms)
2025-02-18 06:19:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901444413 ms)
2025-02-18 06:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901444413 ms)
2025-02-18 06:19:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:19:25.401125+00:00', 'runtime': '901440.002', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901444417 ms)
2025-02-18 06:19:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:18:25.402782+00:00', 'runtime': '901380.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.585', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901444417 ms)
2025-02-18 06:20:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901504416 ms)
2025-02-18 06:20:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901504417 ms)
2025-02-18 06:20:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:20:25.400763+00:00', 'runtime': '901500.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901504421 ms)
2025-02-18 06:20:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:19:25.401125+00:00', 'runtime': '901440.002', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1595.9', 'ahl_t': '1595.9', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901504421 ms)
2025-02-18 06:21:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901564404 ms)
2025-02-18 06:21:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901564404 ms)
2025-02-18 06:21:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:21:25.400639+00:00', 'runtime': '901560.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901564408 ms)
2025-02-18 06:21:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:20:25.400763+00:00', 'runtime': '901500.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901564409 ms)
2025-02-18 06:21:48.413 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901586232 ms)
2025-02-18 06:21:48.413 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901586232 ms)
2025-02-18 06:21:48.413 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41990)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901586233 ms)
2025-02-18 06:22:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901624408 ms)
2025-02-18 06:22:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901624408 ms)
2025-02-18 06:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:22:25.402844+00:00', 'runtime': '901620.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901624412 ms)
2025-02-18 06:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:21:25.400639+00:00', 'runtime': '901560.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901624413 ms)
2025-02-18 06:22:35.522 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901633341 ms)
2025-02-18 06:22:35.522 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901633342 ms)
2025-02-18 06:22:35.523 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42040)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901633342 ms)
2025-02-18 06:22:35.799 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+901633618 ms)
2025-02-18 06:22:35.800 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+901633619 ms)
2025-02-18 06:22:35.801 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:22:25.400202+00:00'} (T+901633620 ms)
2025-02-18 06:23:08.642 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901666461 ms)
2025-02-18 06:23:08.642 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901666462 ms)
2025-02-18 06:23:08.643 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42076)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901666462 ms)
2025-02-18 06:23:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901684411 ms)
2025-02-18 06:23:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901684411 ms)
2025-02-18 06:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:23:25.402628+00:00', 'runtime': '901680.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901684415 ms)
2025-02-18 06:23:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:22:25.402844+00:00', 'runtime': '901620.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901684415 ms)
2025-02-18 06:23:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+901693421 ms)
2025-02-18 06:23:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+901693422 ms)
2025-02-18 06:23:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:23:25.399991+00:00'} (T+901693423 ms)
2025-02-18 06:24:17.822 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901735641 ms)
2025-02-18 06:24:17.822 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901735641 ms)
2025-02-18 06:24:17.822 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42150)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901735642 ms)
2025-02-18 06:24:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901744412 ms)
2025-02-18 06:24:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901744413 ms)
2025-02-18 06:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:24:25.400202+00:00', 'runtime': '901740.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901744416 ms)
2025-02-18 06:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:23:25.402628+00:00', 'runtime': '901680.003', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901744417 ms)
2025-02-18 06:25:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901804416 ms)
2025-02-18 06:25:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901804417 ms)
2025-02-18 06:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:25:25.400772+00:00', 'runtime': '901800.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901804420 ms)
2025-02-18 06:25:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:24:25.400202+00:00', 'runtime': '901740.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901804421 ms)
2025-02-18 06:26:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901864401 ms)
2025-02-18 06:26:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901864402 ms)
2025-02-18 06:26:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:26:25.400224+00:00', 'runtime': '901860.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901864403 ms)
2025-02-18 06:26:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:25:25.400772+00:00', 'runtime': '901800.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901864403 ms)
2025-02-18 06:26:36.052 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901873871 ms)
2025-02-18 06:26:36.052 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901873871 ms)
2025-02-18 06:26:36.052 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42306)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901873872 ms)
2025-02-18 06:27:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901924402 ms)
2025-02-18 06:27:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901924403 ms)
2025-02-18 06:27:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:27:25.400176+00:00', 'runtime': '901920.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901924404 ms)
2025-02-18 06:27:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:26:25.400224+00:00', 'runtime': '901860.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901924404 ms)
2025-02-18 06:28:11.262 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+901969082 ms)
2025-02-18 06:28:11.263 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+901969082 ms)
2025-02-18 06:28:11.263 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42408)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+901969082 ms)
2025-02-18 06:28:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+901984404 ms)
2025-02-18 06:28:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+901984405 ms)
2025-02-18 06:28:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:28:25.400216+00:00', 'runtime': '901980.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901984406 ms)
2025-02-18 06:28:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:27:25.400176+00:00', 'runtime': '901920.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+901984406 ms)
2025-02-18 06:29:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902044406 ms)
2025-02-18 06:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902044407 ms)
2025-02-18 06:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:29:25.400208+00:00', 'runtime': '902040.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902044408 ms)
2025-02-18 06:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:28:25.400216+00:00', 'runtime': '901980.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902044408 ms)
2025-02-18 06:30:09.483 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902087303 ms)
2025-02-18 06:30:09.484 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902087303 ms)
2025-02-18 06:30:09.484 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42534)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902087303 ms)
2025-02-18 06:30:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902104409 ms)
2025-02-18 06:30:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902104410 ms)
2025-02-18 06:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:30:25.400169+00:00', 'runtime': '902100.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902104411 ms)
2025-02-18 06:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:29:25.400208+00:00', 'runtime': '902040.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902104411 ms)
2025-02-18 06:31:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902164411 ms)
2025-02-18 06:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902164412 ms)
2025-02-18 06:31:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:31:25.400169+00:00', 'runtime': '902160.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902164413 ms)
2025-02-18 06:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:30:25.400169+00:00', 'runtime': '902100.001', 'adc_vb_f': '12.598', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902164413 ms)
2025-02-18 06:32:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902224412 ms)
2025-02-18 06:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902224413 ms)
2025-02-18 06:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:32:25.400165+00:00', 'runtime': '902220.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.138', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902224414 ms)
2025-02-18 06:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:31:25.400169+00:00', 'runtime': '902160.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902224414 ms)
2025-02-18 06:33:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902284414 ms)
2025-02-18 06:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902284414 ms)
2025-02-18 06:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:33:25.400173+00:00', 'runtime': '902280.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902284415 ms)
2025-02-18 06:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:32:25.400165+00:00', 'runtime': '902220.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.138', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902284416 ms)
2025-02-18 06:34:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902344415 ms)
2025-02-18 06:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902344416 ms)
2025-02-18 06:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:34:25.400173+00:00', 'runtime': '902340.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.147', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902344417 ms)
2025-02-18 06:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:33:25.400173+00:00', 'runtime': '902280.001', 'adc_vb_f': '12.595', 'adc_va_f': '1.135', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.0', 'ahl_t': '1596.0', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902344417 ms)
2025-02-18 06:34:35.720 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902353539 ms)
2025-02-18 06:34:35.720 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902353539 ms)
2025-02-18 06:34:35.721 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:34:25.400333+00:00'} (T+902353541 ms)
2025-02-18 06:34:54.934 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902372753 ms)
2025-02-18 06:34:54.934 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902372754 ms)
2025-02-18 06:34:54.935 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42834)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902372754 ms)
2025-02-18 06:34:58.014 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902375833 ms)
2025-02-18 06:34:58.014 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902375834 ms)
2025-02-18 06:34:58.015 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42838)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902375834 ms)
2025-02-18 06:35:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902404419 ms)
2025-02-18 06:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902404420 ms)
2025-02-18 06:35:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:35:25.400173+00:00', 'runtime': '902400.001', 'adc_vb_f': '12.589', 'adc_va_f': '1.151', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902404421 ms)
2025-02-18 06:35:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:34:25.400173+00:00', 'runtime': '902340.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.147', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902404422 ms)
2025-02-18 06:35:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902413418 ms)
2025-02-18 06:35:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902413419 ms)
2025-02-18 06:35:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:35:25.400085+00:00'} (T+902413420 ms)
2025-02-18 06:36:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902464403 ms)
2025-02-18 06:36:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902464404 ms)
2025-02-18 06:36:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:36:25.400162+00:00', 'runtime': '902460.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.151', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902464405 ms)
2025-02-18 06:36:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:35:25.400173+00:00', 'runtime': '902400.001', 'adc_vb_f': '12.589', 'adc_va_f': '1.151', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.582', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902464406 ms)
2025-02-18 06:36:36.702 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902474521 ms)
2025-02-18 06:36:36.702 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902474522 ms)
2025-02-18 06:36:36.703 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:36:25.400201+00:00'} (T+902474523 ms)
2025-02-18 06:37:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902524407 ms)
2025-02-18 06:37:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902524408 ms)
2025-02-18 06:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:37:25.400177+00:00', 'runtime': '902520.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.144', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902524409 ms)
2025-02-18 06:37:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:36:25.400162+00:00', 'runtime': '902460.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.151', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902524410 ms)
2025-02-18 06:37:35.658 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902533478 ms)
2025-02-18 06:37:35.659 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902533478 ms)
2025-02-18 06:37:35.660 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:37:25.400096+00:00'} (T+902533479 ms)
2025-02-18 06:37:47.328 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902545148 ms)
2025-02-18 06:37:47.329 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902545148 ms)
2025-02-18 06:37:47.329 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43028)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902545148 ms)
2025-02-18 06:38:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902584412 ms)
2025-02-18 06:38:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902584412 ms)
2025-02-18 06:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:38:25.400187+00:00', 'runtime': '902580.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.157', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902584413 ms)
2025-02-18 06:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:37:25.400177+00:00', 'runtime': '902520.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.144', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902584414 ms)
2025-02-18 06:38:35.593 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902593413 ms)
2025-02-18 06:38:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902593413 ms)
2025-02-18 06:38:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:38:25.400145+00:00'} (T+902593414 ms)
2025-02-18 06:39:02.497 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902620316 ms)
2025-02-18 06:39:02.497 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902620317 ms)
2025-02-18 06:39:02.498 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43110)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902620317 ms)
2025-02-18 06:39:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902644411 ms)
2025-02-18 06:39:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902644412 ms)
2025-02-18 06:39:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:39:25.400177+00:00', 'runtime': '902640.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.151', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902644413 ms)
2025-02-18 06:39:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:38:25.400187+00:00', 'runtime': '902580.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.157', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902644413 ms)
2025-02-18 06:39:35.683 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902653502 ms)
2025-02-18 06:39:35.683 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902653503 ms)
2025-02-18 06:39:35.684 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:39:25.413127+00:00'} (T+902653504 ms)
2025-02-18 06:39:53.597 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902671416 ms)
2025-02-18 06:39:53.597 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902671416 ms)
2025-02-18 06:39:53.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43166)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902671417 ms)
2025-02-18 06:40:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902704413 ms)
2025-02-18 06:40:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902704414 ms)
2025-02-18 06:40:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:40:25.402760+00:00', 'runtime': '902700.003', 'adc_vb_f': '12.589', 'adc_va_f': '1.154', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902704415 ms)
2025-02-18 06:40:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:39:25.400177+00:00', 'runtime': '902640.001', 'adc_vb_f': '12.592', 'adc_va_f': '1.151', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902704415 ms)
2025-02-18 06:40:35.594 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902713414 ms)
2025-02-18 06:40:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902713414 ms)
2025-02-18 06:40:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:40:25.400200+00:00'} (T+902713415 ms)
2025-02-18 06:41:25.757 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902763576 ms)
2025-02-18 06:41:25.757 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902763576 ms)
2025-02-18 06:41:25.757 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43264)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902763577 ms)
2025-02-18 06:41:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902764415 ms)
2025-02-18 06:41:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902764415 ms)
2025-02-18 06:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:41:25.402837+00:00', 'runtime': '902760.003', 'adc_vb_f': '12.585', 'adc_va_f': '1.166', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902764416 ms)
2025-02-18 06:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:40:25.402760+00:00', 'runtime': '902700.003', 'adc_vb_f': '12.589', 'adc_va_f': '1.154', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902764417 ms)
2025-02-18 06:41:29.837 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902767657 ms)
2025-02-18 06:41:29.838 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902767657 ms)
2025-02-18 06:41:29.838 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43270)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902767657 ms)
2025-02-18 06:41:35.714 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902773533 ms)
2025-02-18 06:41:35.714 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902773534 ms)
2025-02-18 06:41:35.715 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:41:25.400207+00:00'} (T+902773534 ms)
2025-02-18 06:42:12.987 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902810806 ms)
2025-02-18 06:42:12.987 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902810806 ms)
2025-02-18 06:42:12.987 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43318)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902810806 ms)
2025-02-18 06:42:24.106 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902821925 ms)
2025-02-18 06:42:24.106 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902821925 ms)
2025-02-18 06:42:24.106 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43330)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902821926 ms)
2025-02-18 06:42:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902824417 ms)
2025-02-18 06:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902824418 ms)
2025-02-18 06:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:42:25.400154+00:00', 'runtime': '902820.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.157', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902824419 ms)
2025-02-18 06:42:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:41:25.402837+00:00', 'runtime': '902760.003', 'adc_vb_f': '12.585', 'adc_va_f': '1.166', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.589', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902824419 ms)
2025-02-18 06:42:27.187 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902825006 ms)
2025-02-18 06:42:27.187 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902825006 ms)
2025-02-18 06:42:27.187 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43334)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902825006 ms)
2025-02-18 06:42:36.204 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902834023 ms)
2025-02-18 06:42:36.205 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902834024 ms)
2025-02-18 06:42:36.206 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:42:25.400074+00:00'} (T+902834025 ms)
2025-02-18 06:43:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902884418 ms)
2025-02-18 06:43:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902884419 ms)
2025-02-18 06:43:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:43:25.400177+00:00', 'runtime': '902880.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.154', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902884420 ms)
2025-02-18 06:43:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:42:25.400154+00:00', 'runtime': '902820.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.157', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902884420 ms)
2025-02-18 06:43:35.654 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902893474 ms)
2025-02-18 06:43:35.655 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902893474 ms)
2025-02-18 06:43:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:43:25.400111+00:00'} (T+902893477 ms)
2025-02-18 06:43:59.364 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+902917184 ms)
2025-02-18 06:43:59.365 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+902917184 ms)
2025-02-18 06:43:59.365 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43436)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+902917185 ms)
2025-02-18 06:44:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+902944404 ms)
2025-02-18 06:44:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+902944405 ms)
2025-02-18 06:44:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:44:25.400198+00:00', 'runtime': '902940.001', 'adc_vb_f': '12.589', 'adc_va_f': '1.157', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902944406 ms)
2025-02-18 06:44:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:43:25.400177+00:00', 'runtime': '902880.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.154', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+902944406 ms)
2025-02-18 06:44:35.586 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902953405 ms)
2025-02-18 06:44:35.586 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+902953405 ms)
2025-02-18 06:44:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:44:25.400175+00:00'} (T+902953413 ms)
2025-02-18 06:45:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903004406 ms)
2025-02-18 06:45:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903004407 ms)
2025-02-18 06:45:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:45:25.400183+00:00', 'runtime': '903000.001', 'adc_vb_f': '12.589', 'adc_va_f': '1.16', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903004408 ms)
2025-02-18 06:45:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:44:25.400198+00:00', 'runtime': '902940.001', 'adc_vb_f': '12.589', 'adc_va_f': '1.157', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903004408 ms)
2025-02-18 06:46:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903064412 ms)
2025-02-18 06:46:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903064412 ms)
2025-02-18 06:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:46:25.400181+00:00', 'runtime': '903060.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.16', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903064413 ms)
2025-02-18 06:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:45:25.400183+00:00', 'runtime': '903000.001', 'adc_vb_f': '12.589', 'adc_va_f': '1.16', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.579', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903064414 ms)
2025-02-18 06:47:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903124413 ms)
2025-02-18 06:47:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903124414 ms)
2025-02-18 06:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:47:25.400163+00:00', 'runtime': '903120.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.169', 'adc_vl_f': '12.576', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903124415 ms)
2025-02-18 06:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:46:25.400181+00:00', 'runtime': '903060.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.16', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903124415 ms)
2025-02-18 06:47:35.691 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903133511 ms)
2025-02-18 06:47:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903133511 ms)
2025-02-18 06:47:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:47:25.400319+00:00'} (T+903133514 ms)
2025-02-18 06:48:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903184413 ms)
2025-02-18 06:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903184414 ms)
2025-02-18 06:48:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:48:25.400172+00:00', 'runtime': '903180.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.175', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903184414 ms)
2025-02-18 06:48:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:47:25.400163+00:00', 'runtime': '903120.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.169', 'adc_vl_f': '12.576', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.1', 'ahl_t': '1596.1', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903184415 ms)
2025-02-18 06:48:30.425 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+903188244 ms)
2025-02-18 06:49:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903244416 ms)
2025-02-18 06:49:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903244416 ms)
2025-02-18 06:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:49:25.400269+00:00', 'runtime': '903240.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903244417 ms)
2025-02-18 06:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:48:25.400172+00:00', 'runtime': '903180.001', 'adc_vb_f': '12.585', 'adc_va_f': '1.175', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903244417 ms)
2025-02-18 06:50:23.910 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903301730 ms)
2025-02-18 06:50:23.911 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903301730 ms)
2025-02-18 06:50:23.911 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43850)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903301731 ms)
2025-02-18 06:50:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903304417 ms)
2025-02-18 06:50:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903304417 ms)
2025-02-18 06:50:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:50:25.400202+00:00', 'runtime': '903300.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.573', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903304418 ms)
2025-02-18 06:50:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:49:25.400269+00:00', 'runtime': '903240.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.582', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903304418 ms)
2025-02-18 06:50:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903313429 ms)
2025-02-18 06:50:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903313429 ms)
2025-02-18 06:50:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:50:25.400326+00:00'} (T+903313431 ms)
2025-02-18 06:51:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903364418 ms)
2025-02-18 06:51:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903364419 ms)
2025-02-18 06:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:51:25.401131+00:00', 'runtime': '903360.002', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.573', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903364420 ms)
2025-02-18 06:51:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:50:25.400202+00:00', 'runtime': '903300.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.573', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903364420 ms)
2025-02-18 06:51:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903373417 ms)
2025-02-18 06:51:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903373418 ms)
2025-02-18 06:51:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:51:25.400195+00:00'} (T+903373419 ms)
2025-02-18 06:51:58.079 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903395898 ms)
2025-02-18 06:51:58.079 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903395899 ms)
2025-02-18 06:51:58.080 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43954)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903395899 ms)
2025-02-18 06:52:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903424420 ms)
2025-02-18 06:52:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903424421 ms)
2025-02-18 06:52:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:52:25.400158+00:00', 'runtime': '903420.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.573', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903424422 ms)
2025-02-18 06:52:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:51:25.401131+00:00', 'runtime': '903360.002', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.573', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.576', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903424422 ms)
2025-02-18 06:52:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903433427 ms)
2025-02-18 06:52:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903433428 ms)
2025-02-18 06:52:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:52:25.413112+00:00'} (T+903433429 ms)
2025-02-18 06:52:41.221 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903439040 ms)
2025-02-18 06:52:41.221 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903439040 ms)
2025-02-18 06:52:41.222 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44000)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903439041 ms)
2025-02-18 06:53:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903484422 ms)
2025-02-18 06:53:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903484422 ms)
2025-02-18 06:53:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:53:25.400452+00:00', 'runtime': '903480.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.57', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903484423 ms)
2025-02-18 06:53:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:52:25.400158+00:00', 'runtime': '903420.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.573', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903484424 ms)
2025-02-18 06:54:10.399 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903528219 ms)
2025-02-18 06:54:10.400 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903528219 ms)
2025-02-18 06:54:10.400 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44096)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903528220 ms)
2025-02-18 06:54:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903544413 ms)
2025-02-18 06:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903544414 ms)
2025-02-18 06:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:54:25.400786+00:00', 'runtime': '903540.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.57', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903544415 ms)
2025-02-18 06:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:53:25.400452+00:00', 'runtime': '903480.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20137', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.57', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903544415 ms)
2025-02-18 06:54:27.510 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903545329 ms)
2025-02-18 06:54:27.510 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903545329 ms)
2025-02-18 06:54:27.511 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44114)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903545330 ms)
2025-02-18 06:55:12.638 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903590457 ms)
2025-02-18 06:55:12.638 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903590457 ms)
2025-02-18 06:55:12.639 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44162)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903590458 ms)
2025-02-18 06:55:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903604415 ms)
2025-02-18 06:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903604416 ms)
2025-02-18 06:55:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:55:25.400603+00:00', 'runtime': '903600.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.175', 'adc_vl_f': '12.579', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.57', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903604418 ms)
2025-02-18 06:55:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:54:25.400786+00:00', 'runtime': '903540.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.585', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.57', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903604418 ms)
2025-02-18 06:56:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903664414 ms)
2025-02-18 06:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903664415 ms)
2025-02-18 06:56:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:56:25.400191+00:00', 'runtime': '903660.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.573', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.567', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903664416 ms)
2025-02-18 06:56:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:55:25.400603+00:00', 'runtime': '903600.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.175', 'adc_vl_f': '12.579', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.57', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903664416 ms)
2025-02-18 06:56:35.872 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903673691 ms)
2025-02-18 06:56:35.872 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903673691 ms)
2025-02-18 06:56:35.877 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:56:25.413050+00:00'} (T+903673696 ms)
2025-02-18 06:57:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903724419 ms)
2025-02-18 06:57:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903724419 ms)
2025-02-18 06:57:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:57:25.400195+00:00', 'runtime': '903720.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.175', 'adc_vl_f': '12.567', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.564', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903724420 ms)
2025-02-18 06:57:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:56:25.400191+00:00', 'runtime': '903660.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.573', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.567', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903724421 ms)
2025-02-18 06:57:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903733428 ms)
2025-02-18 06:57:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903733429 ms)
2025-02-18 06:57:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:57:25.413124+00:00'} (T+903733430 ms)
2025-02-18 06:58:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903784417 ms)
2025-02-18 06:58:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903784417 ms)
2025-02-18 06:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:58:25.400158+00:00', 'runtime': '903780.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.561', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903784419 ms)
2025-02-18 06:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:57:25.400195+00:00', 'runtime': '903720.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.175', 'adc_vl_f': '12.567', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.564', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903784420 ms)
2025-02-18 06:58:35.688 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903793507 ms)
2025-02-18 06:58:35.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903793508 ms)
2025-02-18 06:58:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:58:25.413120+00:00'} (T+903793509 ms)
2025-02-18 06:59:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903844418 ms)
2025-02-18 06:59:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903844419 ms)
2025-02-18 06:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 06:59:25.402699+00:00', 'runtime': '903840.003', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903844420 ms)
2025-02-18 06:59:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:58:25.400158+00:00', 'runtime': '903780.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.561', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903844420 ms)
2025-02-18 06:59:35.691 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903853510 ms)
2025-02-18 06:59:35.692 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903853511 ms)
2025-02-18 06:59:35.693 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 06:59:25.400107+00:00'} (T+903853512 ms)
2025-02-18 07:00:14.079 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903891898 ms)
2025-02-18 07:00:14.079 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903891898 ms)
2025-02-18 07:00:14.080 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44492)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903891899 ms)
2025-02-18 07:00:18.153 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903895972 ms)
2025-02-18 07:00:18.153 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903895972 ms)
2025-02-18 07:00:18.154 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44498)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903895973 ms)
2025-02-18 07:00:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903904417 ms)
2025-02-18 07:00:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903904418 ms)
2025-02-18 07:00:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:00:25.400099+00:00', 'runtime': '903900.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.561', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903904419 ms)
2025-02-18 07:00:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 06:59:25.402699+00:00', 'runtime': '903840.003', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.564', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903904419 ms)
2025-02-18 07:00:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903913439 ms)
2025-02-18 07:00:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903913440 ms)
2025-02-18 07:00:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:00:25.401144+00:00'} (T+903913441 ms)
2025-02-18 07:01:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+903964419 ms)
2025-02-18 07:01:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+903964419 ms)
2025-02-18 07:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:01:25.402836+00:00', 'runtime': '903960.003', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.555', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903964421 ms)
2025-02-18 07:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:00:25.400099+00:00', 'runtime': '903900.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.561', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.2', 'ahl_t': '1596.2', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '2.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+903964422 ms)
2025-02-18 07:01:35.321 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+903973141 ms)
2025-02-18 07:01:35.322 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+903973141 ms)
2025-02-18 07:01:35.322 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44582)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+903973141 ms)
2025-02-18 07:01:35.698 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903973517 ms)
2025-02-18 07:01:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+903973518 ms)
2025-02-18 07:01:35.700 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:01:25.400190+00:00'} (T+903973519 ms)
2025-02-18 07:02:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904024422 ms)
2025-02-18 07:02:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904024422 ms)
2025-02-18 07:02:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:02:25.402733+00:00', 'runtime': '904020.003', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.552', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904024424 ms)
2025-02-18 07:02:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:01:25.402836+00:00', 'runtime': '903960.003', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.555', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904024424 ms)
2025-02-18 07:02:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+904033422 ms)
2025-02-18 07:02:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+904033422 ms)
2025-02-18 07:02:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:02:25.400125+00:00'} (T+904033424 ms)
2025-02-18 07:03:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904084423 ms)
2025-02-18 07:03:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904084423 ms)
2025-02-18 07:03:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:03:25.400136+00:00', 'runtime': '904080.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.57', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904084424 ms)
2025-02-18 07:03:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:02:25.402733+00:00', 'runtime': '904020.003', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.552', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904084425 ms)
2025-02-18 07:03:42.562 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904100381 ms)
2025-02-18 07:03:42.562 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904100382 ms)
2025-02-18 07:03:42.563 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44718)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904100382 ms)
2025-02-18 07:03:47.653 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904105473 ms)
2025-02-18 07:03:47.654 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904105473 ms)
2025-02-18 07:03:47.654 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44724)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904105474 ms)
2025-02-18 07:04:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904144409 ms)
2025-02-18 07:04:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904144409 ms)
2025-02-18 07:04:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:04:25.400579+00:00', 'runtime': '904140.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.57', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904144410 ms)
2025-02-18 07:04:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:03:25.400136+00:00', 'runtime': '904080.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.57', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904144411 ms)
2025-02-18 07:04:47.806 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904165625 ms)
2025-02-18 07:04:47.806 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904165625 ms)
2025-02-18 07:04:47.806 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44788)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904165625 ms)
2025-02-18 07:05:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904204408 ms)
2025-02-18 07:05:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904204409 ms)
2025-02-18 07:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:05:25.400176+00:00', 'runtime': '904200.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.561', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904204410 ms)
2025-02-18 07:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:04:25.400579+00:00', 'runtime': '904140.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.57', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.582', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904204410 ms)
2025-02-18 07:05:59.965 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904237784 ms)
2025-02-18 07:05:59.965 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904237784 ms)
2025-02-18 07:05:59.965 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44876)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904237785 ms)
2025-02-18 07:06:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904264413 ms)
2025-02-18 07:06:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904264413 ms)
2025-02-18 07:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:06:25.400256+00:00', 'runtime': '904260.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904264414 ms)
2025-02-18 07:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:05:25.400176+00:00', 'runtime': '904200.001', 'adc_vb_f': '12.582', 'adc_va_f': '1.178', 'adc_vl_f': '12.561', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904264416 ms)
2025-02-18 07:06:49.096 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904286915 ms)
2025-02-18 07:06:49.096 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904286915 ms)
2025-02-18 07:06:49.097 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44928)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904286916 ms)
2025-02-18 07:07:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904324413 ms)
2025-02-18 07:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904324413 ms)
2025-02-18 07:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:07:25.400610+00:00', 'runtime': '904320.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904324414 ms)
2025-02-18 07:07:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:06:25.400256+00:00', 'runtime': '904260.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904324415 ms)
2025-02-18 07:08:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904384415 ms)
2025-02-18 07:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904384415 ms)
2025-02-18 07:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:08:25.400934+00:00', 'runtime': '904380.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904384416 ms)
2025-02-18 07:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:07:25.400610+00:00', 'runtime': '904320.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904384417 ms)
2025-02-18 07:09:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904444415 ms)
2025-02-18 07:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904444415 ms)
2025-02-18 07:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:09:25.400168+00:00', 'runtime': '904440.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904444416 ms)
2025-02-18 07:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:08:25.400934+00:00', 'runtime': '904380.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904444417 ms)
2025-02-18 07:09:53.400 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904471219 ms)
2025-02-18 07:09:53.400 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904471219 ms)
2025-02-18 07:09:53.401 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45124)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904471220 ms)
2025-02-18 07:10:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904504413 ms)
2025-02-18 07:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904504414 ms)
2025-02-18 07:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:10:25.400156+00:00', 'runtime': '904500.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904504415 ms)
2025-02-18 07:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:09:25.400168+00:00', 'runtime': '904440.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904504416 ms)
2025-02-18 07:11:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904564414 ms)
2025-02-18 07:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904564415 ms)
2025-02-18 07:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:11:25.400163+00:00', 'runtime': '904560.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904564416 ms)
2025-02-18 07:11:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:10:25.400156+00:00', 'runtime': '904500.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904564416 ms)
2025-02-18 07:12:07.637 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904605456 ms)
2025-02-18 07:12:07.637 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904605456 ms)
2025-02-18 07:12:07.638 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45264)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904605457 ms)
2025-02-18 07:12:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904624415 ms)
2025-02-18 07:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904624415 ms)
2025-02-18 07:12:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:12:25.400204+00:00', 'runtime': '904620.001', 'adc_vb_f': '12.576', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.576', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904624419 ms)
2025-02-18 07:12:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:11:25.400163+00:00', 'runtime': '904560.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.579', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904624419 ms)
2025-02-18 07:12:35.759 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+904633578 ms)
2025-02-18 07:12:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+904633579 ms)
2025-02-18 07:12:35.761 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:12:25.405069+00:00'} (T+904633580 ms)
2025-02-18 07:13:05.776 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904663595 ms)
2025-02-18 07:13:05.776 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904663595 ms)
2025-02-18 07:13:05.776 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45328)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904663596 ms)
2025-02-18 07:13:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904684413 ms)
2025-02-18 07:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904684413 ms)
2025-02-18 07:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:13:25.400151+00:00', 'runtime': '904680.001', 'adc_vb_f': '12.576', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.573', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904684414 ms)
2025-02-18 07:13:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:12:25.400204+00:00', 'runtime': '904620.001', 'adc_vb_f': '12.576', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.576', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904684415 ms)
2025-02-18 07:13:52.905 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904710724 ms)
2025-02-18 07:13:52.905 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904710725 ms)
2025-02-18 07:13:52.906 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45380)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904710725 ms)
2025-02-18 07:14:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904744412 ms)
2025-02-18 07:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904744413 ms)
2025-02-18 07:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:14:25.400270+00:00', 'runtime': '904740.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.573', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904744414 ms)
2025-02-18 07:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:13:25.400151+00:00', 'runtime': '904680.001', 'adc_vb_f': '12.576', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.573', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904744414 ms)
2025-02-18 07:14:39.033 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904756852 ms)
2025-02-18 07:14:39.033 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904756853 ms)
2025-02-18 07:14:39.034 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45428)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904756853 ms)
2025-02-18 07:15:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904804413 ms)
2025-02-18 07:15:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904804414 ms)
2025-02-18 07:15:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:15:25.400258+00:00', 'runtime': '904800.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.573', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904804415 ms)
2025-02-18 07:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:14:25.400270+00:00', 'runtime': '904740.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.549', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.573', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.3', 'ahl_t': '1596.3', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904804415 ms)
2025-02-18 07:15:36.575 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+904814394 ms)
2025-02-18 07:15:36.575 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+904814395 ms)
2025-02-18 07:15:36.577 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:15:25.400340+00:00'} (T+904814396 ms)
2025-02-18 07:16:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904864413 ms)
2025-02-18 07:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904864414 ms)
2025-02-18 07:16:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:16:25.400263+00:00', 'runtime': '904860.001', 'adc_vb_f': '12.567', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904864415 ms)
2025-02-18 07:16:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:15:25.400258+00:00', 'runtime': '904800.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.573', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904864415 ms)
2025-02-18 07:17:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904924416 ms)
2025-02-18 07:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904924416 ms)
2025-02-18 07:17:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:17:25.400176+00:00', 'runtime': '904920.001', 'adc_vb_f': '12.564', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904924417 ms)
2025-02-18 07:17:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:16:25.400263+00:00', 'runtime': '904860.001', 'adc_vb_f': '12.567', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904924418 ms)
2025-02-18 07:17:45.346 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+904943165 ms)
2025-02-18 07:17:45.346 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+904943165 ms)
2025-02-18 07:17:45.346 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45636)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+904943166 ms)
2025-02-18 07:18:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+904984417 ms)
2025-02-18 07:18:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+904984417 ms)
2025-02-18 07:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:18:25.400176+00:00', 'runtime': '904980.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904984418 ms)
2025-02-18 07:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:17:25.400176+00:00', 'runtime': '904920.001', 'adc_vb_f': '12.564', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+904984418 ms)
2025-02-18 07:19:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905044401 ms)
2025-02-18 07:19:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905044402 ms)
2025-02-18 07:19:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:19:25.400152+00:00', 'runtime': '905040.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905044403 ms)
2025-02-18 07:19:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:18:25.400176+00:00', 'runtime': '904980.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905044403 ms)
2025-02-18 07:19:42.564 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905060384 ms)
2025-02-18 07:19:42.565 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905060384 ms)
2025-02-18 07:19:42.565 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45760)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905060385 ms)
2025-02-18 07:20:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905104418 ms)
2025-02-18 07:20:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905104418 ms)
2025-02-18 07:20:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:20:25.400267+00:00', 'runtime': '905100.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905104420 ms)
2025-02-18 07:20:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:19:25.400152+00:00', 'runtime': '905040.001', 'adc_vb_f': '12.579', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905104421 ms)
2025-02-18 07:20:56.726 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905134545 ms)
2025-02-18 07:20:56.726 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905134546 ms)
2025-02-18 07:20:56.727 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45842)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905134546 ms)
2025-02-18 07:21:26.581 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905164401 ms)
2025-02-18 07:21:26.582 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905164401 ms)
2025-02-18 07:21:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:21:25.400181+00:00', 'runtime': '905160.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.567', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905164402 ms)
2025-02-18 07:21:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:20:25.400267+00:00', 'runtime': '905100.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.57', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905164403 ms)
2025-02-18 07:21:46.856 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905184675 ms)
2025-02-18 07:21:46.856 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905184676 ms)
2025-02-18 07:21:46.857 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45894)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905184676 ms)
2025-02-18 07:22:19.946 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905217765 ms)
2025-02-18 07:22:19.946 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905217766 ms)
2025-02-18 07:22:19.947 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45930)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905217766 ms)
2025-02-18 07:22:26.582 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905224402 ms)
2025-02-18 07:22:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905224402 ms)
2025-02-18 07:22:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:22:25.400205+00:00', 'runtime': '905220.001', 'adc_vb_f': '12.564', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.567', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905224403 ms)
2025-02-18 07:22:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:21:25.400181+00:00', 'runtime': '905160.001', 'adc_vb_f': '12.573', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.567', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905224404 ms)
2025-02-18 07:23:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905284417 ms)
2025-02-18 07:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905284417 ms)
2025-02-18 07:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:23:25.400190+00:00', 'runtime': '905280.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.564', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905284418 ms)
2025-02-18 07:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:22:25.400205+00:00', 'runtime': '905220.001', 'adc_vb_f': '12.564', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.567', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905284419 ms)
2025-02-18 07:23:36.515 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905294334 ms)
2025-02-18 07:23:36.515 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905294335 ms)
2025-02-18 07:23:36.517 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:23:25.400309+00:00'} (T+905294336 ms)
2025-02-18 07:23:55.194 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905313013 ms)
2025-02-18 07:23:55.194 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905313014 ms)
2025-02-18 07:23:55.195 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46030)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905313014 ms)
2025-02-18 07:24:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905344416 ms)
2025-02-18 07:24:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905344416 ms)
2025-02-18 07:24:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:24:25.400265+00:00', 'runtime': '905340.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905344417 ms)
2025-02-18 07:24:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:23:25.400190+00:00', 'runtime': '905280.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.564', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905344418 ms)
2025-02-18 07:24:35.705 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905353525 ms)
2025-02-18 07:24:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905353525 ms)
2025-02-18 07:24:35.710 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:24:25.400322+00:00'} (T+905353530 ms)
2025-02-18 07:25:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905404403 ms)
2025-02-18 07:25:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905404403 ms)
2025-02-18 07:25:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:25:25.400177+00:00', 'runtime': '905400.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905404404 ms)
2025-02-18 07:25:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:24:25.400265+00:00', 'runtime': '905340.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905404404 ms)
2025-02-18 07:25:35.593 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905413412 ms)
2025-02-18 07:25:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905413412 ms)
2025-02-18 07:25:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:25:25.400100+00:00'} (T+905413420 ms)
2025-02-18 07:25:42.383 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905420202 ms)
2025-02-18 07:25:42.383 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905420202 ms)
2025-02-18 07:25:42.383 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46148)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905420203 ms)
2025-02-18 07:26:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905464403 ms)
2025-02-18 07:26:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905464404 ms)
2025-02-18 07:26:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:26:25.400289+00:00', 'runtime': '905460.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905464405 ms)
2025-02-18 07:26:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:25:25.400177+00:00', 'runtime': '905400.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905464405 ms)
2025-02-18 07:26:35.590 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905473409 ms)
2025-02-18 07:26:35.590 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905473410 ms)
2025-02-18 07:26:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:26:25.400284+00:00'} (T+905473414 ms)
2025-02-18 07:26:41.534 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905479353 ms)
2025-02-18 07:26:41.535 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905479354 ms)
2025-02-18 07:26:41.535 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46220)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905479354 ms)
2025-02-18 07:27:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905524406 ms)
2025-02-18 07:27:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905524407 ms)
2025-02-18 07:27:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:27:25.400200+00:00', 'runtime': '905520.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905524408 ms)
2025-02-18 07:27:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:26:25.400289+00:00', 'runtime': '905460.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.481', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905524408 ms)
2025-02-18 07:27:35.590 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905533410 ms)
2025-02-18 07:27:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905533410 ms)
2025-02-18 07:27:35.592 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:27:25.400218+00:00'} (T+905533412 ms)
2025-02-18 07:27:36.704 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905534523 ms)
2025-02-18 07:27:36.704 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905534523 ms)
2025-02-18 07:27:36.707 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46280)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905534526 ms)
2025-02-18 07:27:38.785 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905536604 ms)
2025-02-18 07:27:38.785 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905536605 ms)
2025-02-18 07:27:38.786 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46284)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905536605 ms)
2025-02-18 07:28:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905584408 ms)
2025-02-18 07:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905584408 ms)
2025-02-18 07:28:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:28:25.400206+00:00', 'runtime': '905580.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905584409 ms)
2025-02-18 07:28:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:27:25.400200+00:00', 'runtime': '905520.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905584409 ms)
2025-02-18 07:29:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905644409 ms)
2025-02-18 07:29:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905644410 ms)
2025-02-18 07:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:29:25.400183+00:00', 'runtime': '905640.001', 'adc_vb_f': '12.561', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905644411 ms)
2025-02-18 07:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:28:25.400206+00:00', 'runtime': '905580.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.4', 'ahl_t': '1596.4', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905644411 ms)
2025-02-18 07:30:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905704410 ms)
2025-02-18 07:30:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905704410 ms)
2025-02-18 07:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:30:25.400163+00:00', 'runtime': '905700.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905704411 ms)
2025-02-18 07:30:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:29:25.400183+00:00', 'runtime': '905640.001', 'adc_vb_f': '12.561', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905704412 ms)
2025-02-18 07:30:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905713419 ms)
2025-02-18 07:30:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+905713420 ms)
2025-02-18 07:30:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:30:25.400327+00:00'} (T+905713422 ms)
2025-02-18 07:31:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905764413 ms)
2025-02-18 07:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905764414 ms)
2025-02-18 07:31:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:31:25.400172+00:00', 'runtime': '905760.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905764415 ms)
2025-02-18 07:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:30:25.400163+00:00', 'runtime': '905700.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905764415 ms)
2025-02-18 07:31:35.137 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905772957 ms)
2025-02-18 07:31:35.138 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905772957 ms)
2025-02-18 07:31:35.138 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46536)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905772958 ms)
2025-02-18 07:32:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905824412 ms)
2025-02-18 07:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905824413 ms)
2025-02-18 07:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:32:25.400173+00:00', 'runtime': '905820.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905824414 ms)
2025-02-18 07:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:31:25.400172+00:00', 'runtime': '905760.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905824414 ms)
2025-02-18 07:33:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905884415 ms)
2025-02-18 07:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905884416 ms)
2025-02-18 07:33:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:33:25.400180+00:00', 'runtime': '905880.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905884417 ms)
2025-02-18 07:33:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:32:25.400173+00:00', 'runtime': '905820.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905884417 ms)
2025-02-18 07:34:08.412 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905926231 ms)
2025-02-18 07:34:08.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905926231 ms)
2025-02-18 07:34:08.413 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46698)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905926232 ms)
2025-02-18 07:34:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+905944417 ms)
2025-02-18 07:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+905944417 ms)
2025-02-18 07:34:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:34:25.400174+00:00', 'runtime': '905940.001', 'adc_vb_f': '12.561', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905944418 ms)
2025-02-18 07:34:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:33:25.400180+00:00', 'runtime': '905880.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+905944419 ms)
2025-02-18 07:34:28.513 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+905946332 ms)
2025-02-18 07:34:28.513 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+905946333 ms)
2025-02-18 07:34:28.514 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46720)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+905946333 ms)
2025-02-18 07:35:25.574 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906003394 ms)
2025-02-18 07:35:25.575 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906003394 ms)
2025-02-18 07:35:25.577 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:35:25.400153+00:00', 'runtime': '906000.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906003396 ms)
2025-02-18 07:35:25.578 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:34:25.400174+00:00', 'runtime': '905940.001', 'adc_vb_f': '12.561', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906003397 ms)
2025-02-18 07:36:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906064406 ms)
2025-02-18 07:36:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906064406 ms)
2025-02-18 07:36:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:36:25.400830+00:00', 'runtime': '906060.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906064409 ms)
2025-02-18 07:36:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:35:25.400153+00:00', 'runtime': '906000.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906064410 ms)
2025-02-18 07:37:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906124407 ms)
2025-02-18 07:37:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906124407 ms)
2025-02-18 07:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:37:25.400618+00:00', 'runtime': '906120.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906124411 ms)
2025-02-18 07:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:36:25.400830+00:00', 'runtime': '906060.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.561', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906124412 ms)
2025-02-18 07:37:34.814 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906132633 ms)
2025-02-18 07:37:34.814 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906132633 ms)
2025-02-18 07:37:34.814 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46926)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906132634 ms)
2025-02-18 07:37:39.904 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906137723 ms)
2025-02-18 07:37:39.904 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906137723 ms)
2025-02-18 07:37:39.904 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46932)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906137724 ms)
2025-02-18 07:38:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906184408 ms)
2025-02-18 07:38:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906184408 ms)
2025-02-18 07:38:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:38:25.400185+00:00', 'runtime': '906180.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906184412 ms)
2025-02-18 07:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:37:25.400618+00:00', 'runtime': '906120.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906184413 ms)
2025-02-18 07:38:54.061 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906211880 ms)
2025-02-18 07:38:54.061 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906211880 ms)
2025-02-18 07:38:54.061 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47014)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906211881 ms)
2025-02-18 07:39:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906244409 ms)
2025-02-18 07:39:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906244409 ms)
2025-02-18 07:39:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:39:25.400676+00:00', 'runtime': '906240.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906244411 ms)
2025-02-18 07:39:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:38:25.400185+00:00', 'runtime': '906180.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906244412 ms)
2025-02-18 07:40:25.637 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906303456 ms)
2025-02-18 07:40:25.638 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906303457 ms)
2025-02-18 07:40:25.642 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:40:25.400931+00:00', 'runtime': '906300.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906303461 ms)
2025-02-18 07:40:25.642 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:39:25.400676+00:00', 'runtime': '906240.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906303461 ms)
2025-02-18 07:41:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906364408 ms)
2025-02-18 07:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906364408 ms)
2025-02-18 07:41:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:41:25.402795+00:00', 'runtime': '906360.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906364409 ms)
2025-02-18 07:41:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:40:25.400931+00:00', 'runtime': '906300.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906364410 ms)
2025-02-18 07:42:18.373 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906416193 ms)
2025-02-18 07:42:18.374 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906416193 ms)
2025-02-18 07:42:18.374 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47228)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906416193 ms)
2025-02-18 07:42:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906424410 ms)
2025-02-18 07:42:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906424410 ms)
2025-02-18 07:42:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:42:25.400695+00:00', 'runtime': '906420.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906424412 ms)
2025-02-18 07:42:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:41:25.402795+00:00', 'runtime': '906360.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906424412 ms)
2025-02-18 07:43:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906484411 ms)
2025-02-18 07:43:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906484411 ms)
2025-02-18 07:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:43:25.400199+00:00', 'runtime': '906480.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906484413 ms)
2025-02-18 07:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:42:25.400695+00:00', 'runtime': '906420.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.5', 'ahl_t': '1596.5', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906484413 ms)
2025-02-18 07:43:31.533 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906489352 ms)
2025-02-18 07:43:31.534 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906489353 ms)
2025-02-18 07:43:31.535 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47304)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906489354 ms)
2025-02-18 07:44:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906544410 ms)
2025-02-18 07:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906544411 ms)
2025-02-18 07:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:44:25.400770+00:00', 'runtime': '906540.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906544415 ms)
2025-02-18 07:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:43:25.400199+00:00', 'runtime': '906480.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906544415 ms)
2025-02-18 07:45:02.719 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906580538 ms)
2025-02-18 07:45:02.719 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906580539 ms)
2025-02-18 07:45:02.720 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47402)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906580539 ms)
2025-02-18 07:45:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906604410 ms)
2025-02-18 07:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906604410 ms)
2025-02-18 07:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:45:25.400169+00:00', 'runtime': '906600.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906604413 ms)
2025-02-18 07:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:44:25.400770+00:00', 'runtime': '906540.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906604414 ms)
2025-02-18 07:46:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906664409 ms)
2025-02-18 07:46:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906664409 ms)
2025-02-18 07:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:46:25.402792+00:00', 'runtime': '906660.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906664413 ms)
2025-02-18 07:46:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:45:25.400169+00:00', 'runtime': '906600.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906664414 ms)
2025-02-18 07:46:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+906673427 ms)
2025-02-18 07:46:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+906673427 ms)
2025-02-18 07:46:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:46:25.400130+00:00'} (T+906673429 ms)
2025-02-18 07:47:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906724409 ms)
2025-02-18 07:47:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906724410 ms)
2025-02-18 07:47:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:47:25.400166+00:00', 'runtime': '906720.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906724414 ms)
2025-02-18 07:47:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:46:25.402792+00:00', 'runtime': '906660.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906724414 ms)
2025-02-18 07:47:55.013 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906752832 ms)
2025-02-18 07:47:55.013 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906752833 ms)
2025-02-18 07:47:55.014 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47596)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906752833 ms)
2025-02-18 07:48:25.133 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906782953 ms)
2025-02-18 07:48:25.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906782953 ms)
2025-02-18 07:48:25.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47628)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906782953 ms)
2025-02-18 07:48:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906784411 ms)
2025-02-18 07:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906784411 ms)
2025-02-18 07:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:48:25.400698+00:00', 'runtime': '906780.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906784412 ms)
2025-02-18 07:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:47:25.400166+00:00', 'runtime': '906720.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906784413 ms)
2025-02-18 07:49:10.260 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906828079 ms)
2025-02-18 07:49:10.260 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906828079 ms)
2025-02-18 07:49:10.260 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47676)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906828080 ms)
2025-02-18 07:49:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906844413 ms)
2025-02-18 07:49:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906844413 ms)
2025-02-18 07:49:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:49:25.400202+00:00', 'runtime': '906840.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906844414 ms)
2025-02-18 07:49:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:48:25.400698+00:00', 'runtime': '906780.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906844415 ms)
2025-02-18 07:49:48.351 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906866170 ms)
2025-02-18 07:49:48.351 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906866171 ms)
2025-02-18 07:49:48.352 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47718)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906866171 ms)
2025-02-18 07:50:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906904411 ms)
2025-02-18 07:50:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906904412 ms)
2025-02-18 07:50:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:50:25.400242+00:00', 'runtime': '906900.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906904413 ms)
2025-02-18 07:50:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:49:25.400202+00:00', 'runtime': '906840.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906904413 ms)
2025-02-18 07:50:33.492 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+906911311 ms)
2025-02-18 07:50:33.492 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+906911312 ms)
2025-02-18 07:50:33.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47764)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+906911312 ms)
2025-02-18 07:50:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+906913429 ms)
2025-02-18 07:50:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+906913429 ms)
2025-02-18 07:50:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:50:25.400347+00:00'} (T+906913431 ms)
2025-02-18 07:51:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+906964409 ms)
2025-02-18 07:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+906964409 ms)
2025-02-18 07:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:51:25.400177+00:00', 'runtime': '906960.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906964410 ms)
2025-02-18 07:51:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:50:25.400242+00:00', 'runtime': '906900.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+906964411 ms)
2025-02-18 07:51:35.695 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+906973514 ms)
2025-02-18 07:51:35.696 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+906973515 ms)
2025-02-18 07:51:35.697 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:51:25.400179+00:00'} (T+906973516 ms)
2025-02-18 07:52:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907024410 ms)
2025-02-18 07:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907024411 ms)
2025-02-18 07:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:52:25.400181+00:00', 'runtime': '907020.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907024412 ms)
2025-02-18 07:52:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:51:25.400177+00:00', 'runtime': '906960.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907024412 ms)
2025-02-18 07:52:54.751 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907052570 ms)
2025-02-18 07:52:54.751 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907052571 ms)
2025-02-18 07:52:54.752 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47918)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907052571 ms)
2025-02-18 07:53:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907084410 ms)
2025-02-18 07:53:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907084410 ms)
2025-02-18 07:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:53:25.400217+00:00', 'runtime': '907080.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907084412 ms)
2025-02-18 07:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:52:25.400181+00:00', 'runtime': '907020.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907084412 ms)
2025-02-18 07:54:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907144411 ms)
2025-02-18 07:54:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907144411 ms)
2025-02-18 07:54:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:54:25.400193+00:00', 'runtime': '907140.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907144412 ms)
2025-02-18 07:54:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:53:25.400217+00:00', 'runtime': '907080.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20138', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907144413 ms)
2025-02-18 07:54:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907153425 ms)
2025-02-18 07:54:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907153426 ms)
2025-02-18 07:54:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:54:25.405067+00:00'} (T+907153429 ms)
2025-02-18 07:55:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907204411 ms)
2025-02-18 07:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907204411 ms)
2025-02-18 07:55:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:55:25.400155+00:00', 'runtime': '907200.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907204412 ms)
2025-02-18 07:55:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:54:25.400193+00:00', 'runtime': '907140.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907204413 ms)
2025-02-18 07:55:37.021 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907214840 ms)
2025-02-18 07:55:37.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907214840 ms)
2025-02-18 07:55:37.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48090)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907214841 ms)
2025-02-18 07:56:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907264410 ms)
2025-02-18 07:56:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907264410 ms)
2025-02-18 07:56:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:56:25.400472+00:00', 'runtime': '907260.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907264411 ms)
2025-02-18 07:56:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:55:25.400155+00:00', 'runtime': '907200.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907264411 ms)
2025-02-18 07:57:07.180 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907305000 ms)
2025-02-18 07:57:07.181 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907305000 ms)
2025-02-18 07:57:07.181 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48196)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907305000 ms)
2025-02-18 07:57:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907324409 ms)
2025-02-18 07:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907324410 ms)
2025-02-18 07:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:57:25.400264+00:00', 'runtime': '907320.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907324411 ms)
2025-02-18 07:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:56:25.400472+00:00', 'runtime': '907260.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.6', 'ahl_t': '1596.6', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907324412 ms)
2025-02-18 07:58:24.330 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907382150 ms)
2025-02-18 07:58:24.331 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907382150 ms)
2025-02-18 07:58:24.331 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48276)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907382150 ms)
2025-02-18 07:58:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907384410 ms)
2025-02-18 07:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907384410 ms)
2025-02-18 07:58:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:58:25.400191+00:00', 'runtime': '907380.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907384412 ms)
2025-02-18 07:58:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:57:25.400264+00:00', 'runtime': '907320.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907384413 ms)
2025-02-18 07:58:28.411 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907386231 ms)
2025-02-18 07:58:28.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907386231 ms)
2025-02-18 07:58:28.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48282)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907386231 ms)
2025-02-18 07:58:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907393426 ms)
2025-02-18 07:58:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907393426 ms)
2025-02-18 07:58:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:58:25.400334+00:00'} (T+907393427 ms)
2025-02-18 07:59:16.550 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907434369 ms)
2025-02-18 07:59:16.550 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907434370 ms)
2025-02-18 07:59:16.551 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48336)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907434370 ms)
2025-02-18 07:59:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907444410 ms)
2025-02-18 07:59:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907444410 ms)
2025-02-18 07:59:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 07:59:25.401121+00:00', 'runtime': '907440.002', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907444414 ms)
2025-02-18 07:59:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:58:25.400191+00:00', 'runtime': '907380.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907444414 ms)
2025-02-18 07:59:35.899 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907453718 ms)
2025-02-18 07:59:35.900 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907453719 ms)
2025-02-18 07:59:35.901 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 07:59:25.400209+00:00'} (T+907453720 ms)
2025-02-18 08:00:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907504411 ms)
2025-02-18 08:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907504411 ms)
2025-02-18 08:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:00:25.400169+00:00', 'runtime': '907500.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907504412 ms)
2025-02-18 08:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 07:59:25.401121+00:00', 'runtime': '907440.002', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907504412 ms)
2025-02-18 08:00:35.454 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907513274 ms)
2025-02-18 08:00:35.455 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907513274 ms)
2025-02-18 08:00:35.456 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:00:25.400393+00:00'} (T+907513276 ms)
2025-02-18 08:01:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907564410 ms)
2025-02-18 08:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907564410 ms)
2025-02-18 08:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:01:25.400234+00:00', 'runtime': '907560.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907564411 ms)
2025-02-18 08:01:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:00:25.400169+00:00', 'runtime': '907500.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907564412 ms)
2025-02-18 08:01:53.828 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907591647 ms)
2025-02-18 08:01:53.828 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907591648 ms)
2025-02-18 08:01:53.829 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48506)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907591648 ms)
2025-02-18 08:02:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907624411 ms)
2025-02-18 08:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907624412 ms)
2025-02-18 08:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:02:25.400177+00:00', 'runtime': '907620.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907624413 ms)
2025-02-18 08:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:01:25.400234+00:00', 'runtime': '907560.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907624413 ms)
2025-02-18 08:03:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907684411 ms)
2025-02-18 08:03:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907684412 ms)
2025-02-18 08:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:03:25.400181+00:00', 'runtime': '907680.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907684416 ms)
2025-02-18 08:03:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:02:25.400177+00:00', 'runtime': '907620.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907684416 ms)
2025-02-18 08:04:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907744411 ms)
2025-02-18 08:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907744411 ms)
2025-02-18 08:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:04:25.400824+00:00', 'runtime': '907740.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907744415 ms)
2025-02-18 08:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:03:25.400181+00:00', 'runtime': '907680.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907744415 ms)
2025-02-18 08:05:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907804414 ms)
2025-02-18 08:05:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907804415 ms)
2025-02-18 08:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:05:25.400168+00:00', 'runtime': '907800.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907804416 ms)
2025-02-18 08:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:04:25.400824+00:00', 'runtime': '907740.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907804416 ms)
2025-02-18 08:05:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907813432 ms)
2025-02-18 08:05:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907813433 ms)
2025-02-18 08:05:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:05:25.409046+00:00'} (T+907813435 ms)
2025-02-18 08:05:47.164 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907824983 ms)
2025-02-18 08:05:47.164 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907824983 ms)
2025-02-18 08:05:47.164 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48758)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907824983 ms)
2025-02-18 08:06:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907864415 ms)
2025-02-18 08:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907864416 ms)
2025-02-18 08:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:06:25.400504+00:00', 'runtime': '907860.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907864417 ms)
2025-02-18 08:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:05:25.400168+00:00', 'runtime': '907800.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907864417 ms)
2025-02-18 08:06:34.277 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907872096 ms)
2025-02-18 08:06:34.277 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907872097 ms)
2025-02-18 08:06:34.278 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48810)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907872097 ms)
2025-02-18 08:06:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907873427 ms)
2025-02-18 08:06:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907873428 ms)
2025-02-18 08:06:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:06:25.400180+00:00'} (T+907873429 ms)
2025-02-18 08:07:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907924416 ms)
2025-02-18 08:07:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907924417 ms)
2025-02-18 08:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:07:25.402821+00:00', 'runtime': '907920.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907924418 ms)
2025-02-18 08:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:06:25.400504+00:00', 'runtime': '907860.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907924418 ms)
2025-02-18 08:07:35.709 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907933529 ms)
2025-02-18 08:07:35.710 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907933529 ms)
2025-02-18 08:07:35.711 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:07:25.400193+00:00'} (T+907933530 ms)
2025-02-18 08:07:47.466 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+907945286 ms)
2025-02-18 08:07:47.467 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+907945286 ms)
2025-02-18 08:07:47.467 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48890)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+907945286 ms)
2025-02-18 08:08:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+907984415 ms)
2025-02-18 08:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+907984416 ms)
2025-02-18 08:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:08:25.402658+00:00', 'runtime': '907980.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907984417 ms)
2025-02-18 08:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:07:25.402821+00:00', 'runtime': '907920.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+907984417 ms)
2025-02-18 08:08:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907993424 ms)
2025-02-18 08:08:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+907993424 ms)
2025-02-18 08:08:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:08:25.400076+00:00'} (T+907993425 ms)
2025-02-18 08:09:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908044417 ms)
2025-02-18 08:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908044418 ms)
2025-02-18 08:09:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:09:25.402826+00:00', 'runtime': '908040.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908044419 ms)
2025-02-18 08:09:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:08:25.402658+00:00', 'runtime': '907980.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908044419 ms)
2025-02-18 08:10:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908104416 ms)
2025-02-18 08:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908104416 ms)
2025-02-18 08:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:10:25.400769+00:00', 'runtime': '908100.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908104417 ms)
2025-02-18 08:10:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:09:25.402826+00:00', 'runtime': '908040.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.7', 'ahl_t': '1596.7', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908104418 ms)
2025-02-18 08:11:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908164417 ms)
2025-02-18 08:11:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908164417 ms)
2025-02-18 08:11:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:11:25.400582+00:00', 'runtime': '908160.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908164419 ms)
2025-02-18 08:11:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:10:25.400769+00:00', 'runtime': '908100.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.382', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908164419 ms)
2025-02-18 08:12:19.879 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908217698 ms)
2025-02-18 08:12:19.879 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908217699 ms)
2025-02-18 08:12:19.880 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49178)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908217699 ms)
2025-02-18 08:12:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908224415 ms)
2025-02-18 08:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908224415 ms)
2025-02-18 08:12:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:12:25.402725+00:00', 'runtime': '908220.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908224416 ms)
2025-02-18 08:12:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:11:25.400582+00:00', 'runtime': '908160.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908224417 ms)
2025-02-18 08:12:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908233423 ms)
2025-02-18 08:12:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908233424 ms)
2025-02-18 08:12:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:12:25.400147+00:00'} (T+908233425 ms)
2025-02-18 08:13:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908284414 ms)
2025-02-18 08:13:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908284414 ms)
2025-02-18 08:13:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:13:25.402786+00:00', 'runtime': '908280.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908284415 ms)
2025-02-18 08:13:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:12:25.402725+00:00', 'runtime': '908220.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908284416 ms)
2025-02-18 08:13:35.703 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908293522 ms)
2025-02-18 08:13:35.704 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908293523 ms)
2025-02-18 08:13:35.705 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:13:25.400193+00:00'} (T+908293524 ms)
2025-02-18 08:14:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908344414 ms)
2025-02-18 08:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908344414 ms)
2025-02-18 08:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:14:25.402670+00:00', 'runtime': '908340.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908344415 ms)
2025-02-18 08:14:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:13:25.402786+00:00', 'runtime': '908280.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908344416 ms)
2025-02-18 08:14:32.108 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908349928 ms)
2025-02-18 08:14:32.109 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908349928 ms)
2025-02-18 08:14:32.109 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49320)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908349929 ms)
2025-02-18 08:14:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908353422 ms)
2025-02-18 08:14:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908353423 ms)
2025-02-18 08:14:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:14:25.400097+00:00'} (T+908353424 ms)
2025-02-18 08:15:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908404415 ms)
2025-02-18 08:15:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908404415 ms)
2025-02-18 08:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:15:25.402824+00:00', 'runtime': '908400.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908404416 ms)
2025-02-18 08:15:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:14:25.402670+00:00', 'runtime': '908340.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908404417 ms)
2025-02-18 08:15:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908413434 ms)
2025-02-18 08:15:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908413434 ms)
2025-02-18 08:15:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:15:25.400223+00:00'} (T+908413436 ms)
2025-02-18 08:15:36.186 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+908414005 ms)
2025-02-18 08:15:36.186 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+908414005 ms)
2025-02-18 08:15:36.187 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908414006 ms)
2025-02-18 08:16:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908464417 ms)
2025-02-18 08:16:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908464418 ms)
2025-02-18 08:16:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:16:25.402749+00:00', 'runtime': '908460.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908464419 ms)
2025-02-18 08:16:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:15:25.402824+00:00', 'runtime': '908400.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908464419 ms)
2025-02-18 08:16:35.588 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908473407 ms)
2025-02-18 08:16:35.588 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908473408 ms)
2025-02-18 08:16:35.589 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:16:25.400179+00:00'} (T+908473409 ms)
2025-02-18 08:17:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908524418 ms)
2025-02-18 08:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908524418 ms)
2025-02-18 08:17:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:17:25.402753+00:00', 'runtime': '908520.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908524419 ms)
2025-02-18 08:17:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:16:25.402749+00:00', 'runtime': '908460.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908524420 ms)
2025-02-18 08:17:35.430 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908533249 ms)
2025-02-18 08:17:35.431 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908533250 ms)
2025-02-18 08:17:35.432 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49528)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908533251 ms)
2025-02-18 08:17:35.758 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908533577 ms)
2025-02-18 08:17:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908533578 ms)
2025-02-18 08:17:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:17:25.400184+00:00'} (T+908533578 ms)
2025-02-18 08:18:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908584418 ms)
2025-02-18 08:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908584418 ms)
2025-02-18 08:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:18:25.400187+00:00', 'runtime': '908580.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908584419 ms)
2025-02-18 08:18:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:17:25.402753+00:00', 'runtime': '908520.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.558', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908584420 ms)
2025-02-18 08:18:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908593424 ms)
2025-02-18 08:18:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908593425 ms)
2025-02-18 08:18:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:18:25.400049+00:00'} (T+908593425 ms)
2025-02-18 08:19:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908644406 ms)
2025-02-18 08:19:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908644406 ms)
2025-02-18 08:19:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:19:25.400166+00:00', 'runtime': '908640.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908644407 ms)
2025-02-18 08:19:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:18:25.400187+00:00', 'runtime': '908580.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908644408 ms)
2025-02-18 08:19:35.692 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908653512 ms)
2025-02-18 08:19:35.693 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908653512 ms)
2025-02-18 08:19:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:19:25.400199+00:00'} (T+908653514 ms)
2025-02-18 08:20:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908704409 ms)
2025-02-18 08:20:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908704409 ms)
2025-02-18 08:20:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:20:25.400555+00:00', 'runtime': '908700.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908704410 ms)
2025-02-18 08:20:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:19:25.400166+00:00', 'runtime': '908640.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908704410 ms)
2025-02-18 08:20:35.595 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908713415 ms)
2025-02-18 08:20:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908713415 ms)
2025-02-18 08:20:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:20:25.400108+00:00'} (T+908713417 ms)
2025-02-18 08:21:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908764409 ms)
2025-02-18 08:21:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908764410 ms)
2025-02-18 08:21:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:21:25.400212+00:00', 'runtime': '908760.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908764411 ms)
2025-02-18 08:21:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:20:25.400555+00:00', 'runtime': '908700.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908764411 ms)
2025-02-18 08:21:35.708 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908773528 ms)
2025-02-18 08:21:35.709 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908773528 ms)
2025-02-18 08:21:35.710 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:21:25.400237+00:00'} (T+908773529 ms)
2025-02-18 08:21:57.831 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908795650 ms)
2025-02-18 08:21:57.831 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908795650 ms)
2025-02-18 08:21:57.831 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49810)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908795651 ms)
2025-02-18 08:22:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908824411 ms)
2025-02-18 08:22:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908824411 ms)
2025-02-18 08:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:22:25.400183+00:00', 'runtime': '908820.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908824412 ms)
2025-02-18 08:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:21:25.400212+00:00', 'runtime': '908760.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908824413 ms)
2025-02-18 08:22:35.588 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908833407 ms)
2025-02-18 08:22:35.589 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908833408 ms)
2025-02-18 08:22:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:22:25.400101+00:00'} (T+908833413 ms)
2025-02-18 08:22:42.969 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908840788 ms)
2025-02-18 08:22:42.969 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908840789 ms)
2025-02-18 08:22:42.970 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49858)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908840789 ms)
2025-02-18 08:23:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908884412 ms)
2025-02-18 08:23:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908884413 ms)
2025-02-18 08:23:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:23:25.400164+00:00', 'runtime': '908880.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908884414 ms)
2025-02-18 08:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:22:25.400183+00:00', 'runtime': '908820.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908884414 ms)
2025-02-18 08:23:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908893430 ms)
2025-02-18 08:23:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908893431 ms)
2025-02-18 08:23:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:23:25.405131+00:00'} (T+908893435 ms)
2025-02-18 08:24:26.163 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908943982 ms)
2025-02-18 08:24:26.163 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908943982 ms)
2025-02-18 08:24:26.163 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49968)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908943982 ms)
2025-02-18 08:24:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+908944415 ms)
2025-02-18 08:24:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+908944415 ms)
2025-02-18 08:24:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:24:25.402775+00:00', 'runtime': '908940.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908944417 ms)
2025-02-18 08:24:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:23:25.400164+00:00', 'runtime': '908880.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.8', 'ahl_t': '1596.8', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+908944417 ms)
2025-02-18 08:24:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908953419 ms)
2025-02-18 08:24:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+908953420 ms)
2025-02-18 08:24:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:24:25.400195+00:00'} (T+908953424 ms)
2025-02-18 08:24:46.254 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908964073 ms)
2025-02-18 08:24:46.254 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908964073 ms)
2025-02-18 08:24:46.255 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49992)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908964074 ms)
2025-02-18 08:24:53.314 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+908971133 ms)
2025-02-18 08:24:53.314 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+908971133 ms)
2025-02-18 08:24:53.314 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50000)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+908971133 ms)
2025-02-18 08:25:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909004413 ms)
2025-02-18 08:25:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909004414 ms)
2025-02-18 08:25:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:25:25.400195+00:00', 'runtime': '909000.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909004418 ms)
2025-02-18 08:25:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:24:25.402775+00:00', 'runtime': '908940.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.495', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.555', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909004418 ms)
2025-02-18 08:25:35.732 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909013551 ms)
2025-02-18 08:25:35.732 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909013551 ms)
2025-02-18 08:25:35.734 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:25:25.413123+00:00'} (T+909013553 ms)
2025-02-18 08:26:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909064418 ms)
2025-02-18 08:26:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909064418 ms)
2025-02-18 08:26:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:26:25.402694+00:00', 'runtime': '909060.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909064420 ms)
2025-02-18 08:26:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:25:25.400195+00:00', 'runtime': '909000.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909064420 ms)
2025-02-18 08:26:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909073422 ms)
2025-02-18 08:26:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909073423 ms)
2025-02-18 08:26:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:26:25.400076+00:00'} (T+909073425 ms)
2025-02-18 08:27:02.556 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909100375 ms)
2025-02-18 08:27:02.556 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909100376 ms)
2025-02-18 08:27:02.557 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50150)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909100376 ms)
2025-02-18 08:27:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909124420 ms)
2025-02-18 08:27:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909124420 ms)
2025-02-18 08:27:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:27:25.400176+00:00', 'runtime': '909120.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909124421 ms)
2025-02-18 08:27:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:26:25.402694+00:00', 'runtime': '909060.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909124421 ms)
2025-02-18 08:27:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909133425 ms)
2025-02-18 08:27:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909133426 ms)
2025-02-18 08:27:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:27:25.413124+00:00'} (T+909133427 ms)
2025-02-18 08:28:02.708 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909160527 ms)
2025-02-18 08:28:02.708 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909160527 ms)
2025-02-18 08:28:02.708 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50216)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909160527 ms)
2025-02-18 08:28:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909184423 ms)
2025-02-18 08:28:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909184423 ms)
2025-02-18 08:28:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:28:25.402827+00:00', 'runtime': '909180.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909184425 ms)
2025-02-18 08:28:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:27:25.400176+00:00', 'runtime': '909120.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909184426 ms)
2025-02-18 08:28:52.819 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909210638 ms)
2025-02-18 08:28:52.819 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909210638 ms)
2025-02-18 08:28:52.819 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50272)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909210638 ms)
2025-02-18 08:29:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909244409 ms)
2025-02-18 08:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909244410 ms)
2025-02-18 08:29:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:29:25.400685+00:00', 'runtime': '909240.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909244412 ms)
2025-02-18 08:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:28:25.402827+00:00', 'runtime': '909180.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909244412 ms)
2025-02-18 08:30:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909304409 ms)
2025-02-18 08:30:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909304409 ms)
2025-02-18 08:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:30:25.400190+00:00', 'runtime': '909300.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909304411 ms)
2025-02-18 08:30:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:29:25.400685+00:00', 'runtime': '909240.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909304412 ms)
2025-02-18 08:30:33.019 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909310839 ms)
2025-02-18 08:30:33.020 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909310839 ms)
2025-02-18 08:30:33.020 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50376)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909310840 ms)
2025-02-18 08:30:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909313429 ms)
2025-02-18 08:30:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909313430 ms)
2025-02-18 08:30:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:30:25.413052+00:00'} (T+909313432 ms)
2025-02-18 08:30:38.109 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909315928 ms)
2025-02-18 08:30:38.109 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909315928 ms)
2025-02-18 08:30:38.110 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50384)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909315929 ms)
2025-02-18 08:31:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909364409 ms)
2025-02-18 08:31:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909364410 ms)
2025-02-18 08:31:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:31:25.402831+00:00', 'runtime': '909360.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909364411 ms)
2025-02-18 08:31:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:30:25.400190+00:00', 'runtime': '909300.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.546', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909364411 ms)
2025-02-18 08:31:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909373435 ms)
2025-02-18 08:31:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909373435 ms)
2025-02-18 08:31:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:31:25.400205+00:00'} (T+909373437 ms)
2025-02-18 08:32:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909424411 ms)
2025-02-18 08:32:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909424411 ms)
2025-02-18 08:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:32:25.402797+00:00', 'runtime': '909420.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909424412 ms)
2025-02-18 08:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:31:25.402831+00:00', 'runtime': '909360.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909424413 ms)
2025-02-18 08:32:35.739 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909433558 ms)
2025-02-18 08:32:35.739 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909433558 ms)
2025-02-18 08:32:35.741 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:32:25.400177+00:00'} (T+909433560 ms)
2025-02-18 08:32:50.319 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909448139 ms)
2025-02-18 08:32:50.320 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909448139 ms)
2025-02-18 08:32:50.320 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50526)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909448140 ms)
2025-02-18 08:32:55.412 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909453231 ms)
2025-02-18 08:32:55.412 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909453232 ms)
2025-02-18 08:32:55.413 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50534)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909453232 ms)
2025-02-18 08:33:13.500 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909471319 ms)
2025-02-18 08:33:13.500 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909471320 ms)
2025-02-18 08:33:13.501 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50554)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909471320 ms)
2025-02-18 08:33:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909484413 ms)
2025-02-18 08:33:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909484413 ms)
2025-02-18 08:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:33:25.400162+00:00', 'runtime': '909480.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909484415 ms)
2025-02-18 08:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:32:25.402797+00:00', 'runtime': '909420.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909484416 ms)
2025-02-18 08:33:35.706 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909493526 ms)
2025-02-18 08:33:35.707 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909493526 ms)
2025-02-18 08:33:35.708 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:33:25.400049+00:00'} (T+909493528 ms)
2025-02-18 08:34:13.652 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909531472 ms)
2025-02-18 08:34:13.653 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909531472 ms)
2025-02-18 08:34:13.653 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50620)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909531472 ms)
2025-02-18 08:34:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909544413 ms)
2025-02-18 08:34:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909544413 ms)
2025-02-18 08:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:34:25.400166+00:00', 'runtime': '909540.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909544417 ms)
2025-02-18 08:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:33:25.400162+00:00', 'runtime': '909480.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909544417 ms)
2025-02-18 08:34:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909553419 ms)
2025-02-18 08:34:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909553419 ms)
2025-02-18 08:34:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:34:25.400086+00:00'} (T+909553423 ms)
2025-02-18 08:34:42.779 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909560599 ms)
2025-02-18 08:34:42.780 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909560599 ms)
2025-02-18 08:34:42.780 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50652)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909560599 ms)
2025-02-18 08:35:00.862 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909578681 ms)
2025-02-18 08:35:00.862 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909578682 ms)
2025-02-18 08:35:00.863 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50674)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909578682 ms)
2025-02-18 08:35:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909604415 ms)
2025-02-18 08:35:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909604416 ms)
2025-02-18 08:35:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:35:25.400161+00:00', 'runtime': '909600.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909604417 ms)
2025-02-18 08:35:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:34:25.400166+00:00', 'runtime': '909540.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909604417 ms)
2025-02-18 08:35:34.972 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909612791 ms)
2025-02-18 08:35:34.972 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909612791 ms)
2025-02-18 08:35:34.972 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50712)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909612792 ms)
2025-02-18 08:35:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909613419 ms)
2025-02-18 08:35:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909613420 ms)
2025-02-18 08:35:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:35:25.405140+00:00'} (T+909613424 ms)
2025-02-18 08:36:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909664415 ms)
2025-02-18 08:36:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909664415 ms)
2025-02-18 08:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:36:25.400183+00:00', 'runtime': '909660.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909664416 ms)
2025-02-18 08:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:35:25.400161+00:00', 'runtime': '909600.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909664417 ms)
2025-02-18 08:36:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909673421 ms)
2025-02-18 08:36:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909673421 ms)
2025-02-18 08:36:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:36:25.413123+00:00'} (T+909673426 ms)
2025-02-18 08:36:51.120 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909688940 ms)
2025-02-18 08:36:51.121 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909688940 ms)
2025-02-18 08:36:51.122 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50806)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909688942 ms)
2025-02-18 08:37:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909724413 ms)
2025-02-18 08:37:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909724413 ms)
2025-02-18 08:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:37:25.400173+00:00', 'runtime': '909720.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909724414 ms)
2025-02-18 08:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:36:25.400183+00:00', 'runtime': '909660.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909724415 ms)
2025-02-18 08:37:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909733432 ms)
2025-02-18 08:37:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909733433 ms)
2025-02-18 08:37:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:37:25.413121+00:00'} (T+909733434 ms)
2025-02-18 08:37:55.252 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909753071 ms)
2025-02-18 08:37:55.252 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909753072 ms)
2025-02-18 08:37:55.253 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50876)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909753072 ms)
2025-02-18 08:38:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909784415 ms)
2025-02-18 08:38:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909784415 ms)
2025-02-18 08:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:38:25.402787+00:00', 'runtime': '909780.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909784416 ms)
2025-02-18 08:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:37:25.400173+00:00', 'runtime': '909720.001', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1596.9', 'ahl_t': '1596.9', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909784417 ms)
2025-02-18 08:38:35.605 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909793424 ms)
2025-02-18 08:38:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909793425 ms)
2025-02-18 08:38:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:38:25.400186+00:00'} (T+909793426 ms)
2025-02-18 08:39:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909844413 ms)
2025-02-18 08:39:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909844413 ms)
2025-02-18 08:39:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:39:25.402786+00:00', 'runtime': '909840.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909844414 ms)
2025-02-18 08:39:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:38:25.402787+00:00', 'runtime': '909780.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909844415 ms)
2025-02-18 08:39:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909853436 ms)
2025-02-18 08:39:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909853437 ms)
2025-02-18 08:39:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:39:25.400214+00:00'} (T+909853439 ms)
2025-02-18 08:39:48.492 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909866311 ms)
2025-02-18 08:39:48.492 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909866311 ms)
2025-02-18 08:39:48.492 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50994)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909866312 ms)
2025-02-18 08:40:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909904412 ms)
2025-02-18 08:40:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909904412 ms)
2025-02-18 08:40:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:40:25.402772+00:00', 'runtime': '909900.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909904413 ms)
2025-02-18 08:40:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:39:25.402786+00:00', 'runtime': '909840.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909904414 ms)
2025-02-18 08:40:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909913440 ms)
2025-02-18 08:40:35.621 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909913440 ms)
2025-02-18 08:40:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:40:25.400171+00:00'} (T+909913442 ms)
2025-02-18 08:41:15.684 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+909953503 ms)
2025-02-18 08:41:15.684 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+909953504 ms)
2025-02-18 08:41:15.685 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51088)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+909953504 ms)
2025-02-18 08:41:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+909964416 ms)
2025-02-18 08:41:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+909964417 ms)
2025-02-18 08:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:41:25.403160+00:00', 'runtime': '909960.004', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909964418 ms)
2025-02-18 08:41:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:40:25.402772+00:00', 'runtime': '909900.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+909964418 ms)
2025-02-18 08:41:35.727 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909973546 ms)
2025-02-18 08:41:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+909973547 ms)
2025-02-18 08:41:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:41:25.400248+00:00'} (T+909973549 ms)
2025-02-18 08:42:04.804 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910002623 ms)
2025-02-18 08:42:04.804 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910002623 ms)
2025-02-18 08:42:04.805 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51142)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910002624 ms)
2025-02-18 08:42:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910024415 ms)
2025-02-18 08:42:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910024415 ms)
2025-02-18 08:42:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:42:25.402738+00:00', 'runtime': '910020.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910024416 ms)
2025-02-18 08:42:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:41:25.403160+00:00', 'runtime': '909960.004', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910024416 ms)
2025-02-18 08:42:35.593 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910033412 ms)
2025-02-18 08:42:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910033412 ms)
2025-02-18 08:42:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:42:25.400087+00:00'} (T+910033414 ms)
2025-02-18 08:43:20.957 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910078776 ms)
2025-02-18 08:43:20.957 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910078776 ms)
2025-02-18 08:43:20.957 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51224)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910078776 ms)
2025-02-18 08:43:26.639 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910084458 ms)
2025-02-18 08:43:26.640 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910084459 ms)
2025-02-18 08:43:26.641 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:43:25.402823+00:00', 'runtime': '910080.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910084460 ms)
2025-02-18 08:43:26.641 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:42:25.402738+00:00', 'runtime': '910020.003', 'adc_vb_f': '12.558', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910084460 ms)
2025-02-18 08:43:35.592 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910093412 ms)
2025-02-18 08:43:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910093412 ms)
2025-02-18 08:43:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:43:25.400237+00:00'} (T+910093417 ms)
2025-02-18 08:44:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910144418 ms)
2025-02-18 08:44:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910144418 ms)
2025-02-18 08:44:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:44:25.402759+00:00', 'runtime': '910140.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910144419 ms)
2025-02-18 08:44:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:43:25.402823+00:00', 'runtime': '910080.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.478', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910144420 ms)
2025-02-18 08:45:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910204403 ms)
2025-02-18 08:45:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910204404 ms)
2025-02-18 08:45:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:45:25.400806+00:00', 'runtime': '910200.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910204405 ms)
2025-02-18 08:45:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:44:25.402759+00:00', 'runtime': '910140.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910204405 ms)
2025-02-18 08:46:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910264420 ms)
2025-02-18 08:46:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910264420 ms)
2025-02-18 08:46:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:46:25.400200+00:00', 'runtime': '910260.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910264421 ms)
2025-02-18 08:46:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:45:25.400806+00:00', 'runtime': '910200.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910264422 ms)
2025-02-18 08:46:35.782 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910273601 ms)
2025-02-18 08:46:35.782 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910273602 ms)
2025-02-18 08:46:35.786 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:46:25.413051+00:00'} (T+910273606 ms)
2025-02-18 08:47:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910324406 ms)
2025-02-18 08:47:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910324407 ms)
2025-02-18 08:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:47:25.400174+00:00', 'runtime': '910320.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910324411 ms)
2025-02-18 08:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:46:25.400200+00:00', 'runtime': '910260.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910324411 ms)
2025-02-18 08:47:35.584 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910333403 ms)
2025-02-18 08:47:35.584 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910333404 ms)
2025-02-18 08:47:35.589 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:47:25.400004+00:00'} (T+910333408 ms)
2025-02-18 08:47:57.367 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910355186 ms)
2025-02-18 08:47:57.367 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910355186 ms)
2025-02-18 08:47:57.368 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51530)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910355187 ms)
2025-02-18 08:48:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910384409 ms)
2025-02-18 08:48:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910384409 ms)
2025-02-18 08:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:48:25.400161+00:00', 'runtime': '910380.001', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910384412 ms)
2025-02-18 08:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:47:25.400174+00:00', 'runtime': '910320.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910384413 ms)
2025-02-18 08:48:29.476 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910387296 ms)
2025-02-18 08:48:29.477 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910387296 ms)
2025-02-18 08:48:29.478 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51564)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910387297 ms)
2025-02-18 08:48:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910393429 ms)
2025-02-18 08:48:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910393429 ms)
2025-02-18 08:48:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:48:25.400199+00:00'} (T+910393435 ms)
2025-02-18 08:49:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910444409 ms)
2025-02-18 08:49:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910444409 ms)
2025-02-18 08:49:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:49:25.402758+00:00', 'runtime': '910440.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910444413 ms)
2025-02-18 08:49:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:48:25.400161+00:00', 'runtime': '910380.001', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910444414 ms)
2025-02-18 08:49:35.595 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910453415 ms)
2025-02-18 08:49:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910453415 ms)
2025-02-18 08:49:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:49:25.400183+00:00'} (T+910453417 ms)
2025-02-18 08:50:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910504410 ms)
2025-02-18 08:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910504410 ms)
2025-02-18 08:50:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:50:25.400294+00:00', 'runtime': '910500.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910504414 ms)
2025-02-18 08:50:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:49:25.402758+00:00', 'runtime': '910440.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910504415 ms)
2025-02-18 08:50:35.461 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910513281 ms)
2025-02-18 08:50:35.462 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910513281 ms)
2025-02-18 08:50:35.463 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:50:25.400292+00:00'} (T+910513283 ms)
2025-02-18 08:51:19.766 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910557585 ms)
2025-02-18 08:51:19.766 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910557586 ms)
2025-02-18 08:51:19.767 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51748)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910557586 ms)
2025-02-18 08:51:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910564408 ms)
2025-02-18 08:51:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910564408 ms)
2025-02-18 08:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:51:25.400174+00:00', 'runtime': '910560.001', 'adc_vb_f': '12.552', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910564409 ms)
2025-02-18 08:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:50:25.400294+00:00', 'runtime': '910500.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.0', 'ahl_t': '1597.0', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910564410 ms)
2025-02-18 08:51:35.591 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910573410 ms)
2025-02-18 08:51:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910573411 ms)
2025-02-18 08:51:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:51:25.400361+00:00'} (T+910573415 ms)
2025-02-18 08:52:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910624423 ms)
2025-02-18 08:52:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910624423 ms)
2025-02-18 08:52:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:52:25.400187+00:00', 'runtime': '910620.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910624427 ms)
2025-02-18 08:52:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:51:25.400174+00:00', 'runtime': '910560.001', 'adc_vb_f': '12.552', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910624427 ms)
2025-02-18 08:52:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910633421 ms)
2025-02-18 08:52:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910633422 ms)
2025-02-18 08:52:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:52:25.413124+00:00'} (T+910633424 ms)
2025-02-18 08:53:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910684420 ms)
2025-02-18 08:53:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910684420 ms)
2025-02-18 08:53:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:53:25.400207+00:00', 'runtime': '910680.001', 'adc_vb_f': '12.552', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910684421 ms)
2025-02-18 08:53:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:52:25.400187+00:00', 'runtime': '910620.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910684422 ms)
2025-02-18 08:53:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910693417 ms)
2025-02-18 08:53:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910693418 ms)
2025-02-18 08:53:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:53:25.405151+00:00'} (T+910693422 ms)
2025-02-18 08:54:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910744418 ms)
2025-02-18 08:54:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910744418 ms)
2025-02-18 08:54:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:54:25.402740+00:00', 'runtime': '910740.003', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910744419 ms)
2025-02-18 08:54:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:53:25.400207+00:00', 'runtime': '910680.001', 'adc_vb_f': '12.552', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20139', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910744419 ms)
2025-02-18 08:55:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910804415 ms)
2025-02-18 08:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910804415 ms)
2025-02-18 08:55:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:55:25.400600+00:00', 'runtime': '910800.001', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910804416 ms)
2025-02-18 08:55:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:54:25.402740+00:00', 'runtime': '910740.003', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910804417 ms)
2025-02-18 08:56:12.154 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910849974 ms)
2025-02-18 08:56:12.155 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910849974 ms)
2025-02-18 08:56:12.155 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52070)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910849975 ms)
2025-02-18 08:56:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910864412 ms)
2025-02-18 08:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910864413 ms)
2025-02-18 08:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:56:25.400177+00:00', 'runtime': '910860.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910864414 ms)
2025-02-18 08:56:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:55:25.400600+00:00', 'runtime': '910800.001', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.543', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910864414 ms)
2025-02-18 08:56:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910873433 ms)
2025-02-18 08:56:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910873434 ms)
2025-02-18 08:56:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:56:25.413048+00:00'} (T+910873435 ms)
2025-02-18 08:57:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910924410 ms)
2025-02-18 08:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910924411 ms)
2025-02-18 08:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:57:25.402797+00:00', 'runtime': '910920.003', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910924412 ms)
2025-02-18 08:57:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:56:25.400177+00:00', 'runtime': '910860.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.49', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910924412 ms)
2025-02-18 08:57:35.698 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910933517 ms)
2025-02-18 08:57:35.698 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+910933518 ms)
2025-02-18 08:57:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 08:57:25.400197+00:00'} (T+910933519 ms)
2025-02-18 08:57:59.343 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910957163 ms)
2025-02-18 08:57:59.344 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910957163 ms)
2025-02-18 08:57:59.344 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52186)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910957163 ms)
2025-02-18 08:58:15.433 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+910973252 ms)
2025-02-18 08:58:15.433 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+910973252 ms)
2025-02-18 08:58:15.434 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52204)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+910973253 ms)
2025-02-18 08:58:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+910984413 ms)
2025-02-18 08:58:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+910984413 ms)
2025-02-18 08:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:58:25.402667+00:00', 'runtime': '910980.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910984414 ms)
2025-02-18 08:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:57:25.402797+00:00', 'runtime': '910920.003', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+910984414 ms)
2025-02-18 08:59:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911044409 ms)
2025-02-18 08:59:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911044409 ms)
2025-02-18 08:59:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 08:59:25.400203+00:00', 'runtime': '911040.001', 'adc_vb_f': '12.552', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911044410 ms)
2025-02-18 08:59:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:58:25.402667+00:00', 'runtime': '910980.003', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911044411 ms)
2025-02-18 09:00:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911104407 ms)
2025-02-18 09:00:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911104408 ms)
2025-02-18 09:00:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:00:25.400163+00:00', 'runtime': '911100.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911104409 ms)
2025-02-18 09:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 08:59:25.400203+00:00', 'runtime': '911040.001', 'adc_vb_f': '12.552', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911104411 ms)
2025-02-18 09:00:26.680 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+911104499 ms)
2025-02-18 09:00:26.680 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+911104500 ms)
2025-02-18 09:00:26.681 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52342)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+911104500 ms)
2025-02-18 09:01:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911164403 ms)
2025-02-18 09:01:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911164404 ms)
2025-02-18 09:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:01:25.400197+00:00', 'runtime': '911160.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911164408 ms)
2025-02-18 09:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:00:25.400163+00:00', 'runtime': '911100.001', 'adc_vb_f': '12.555', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911164408 ms)
2025-02-18 09:01:51.869 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+911189688 ms)
2025-02-18 09:01:51.869 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+911189688 ms)
2025-02-18 09:01:51.869 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52434)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+911189689 ms)
2025-02-18 09:02:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911224404 ms)
2025-02-18 09:02:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911224405 ms)
2025-02-18 09:02:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:02:25.400197+00:00', 'runtime': '911220.001', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911224406 ms)
2025-02-18 09:02:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:01:25.400197+00:00', 'runtime': '911160.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911224406 ms)
2025-02-18 09:03:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911284403 ms)
2025-02-18 09:03:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911284404 ms)
2025-02-18 09:03:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:03:25.400201+00:00', 'runtime': '911280.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911284405 ms)
2025-02-18 09:03:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:02:25.400197+00:00', 'runtime': '911220.001', 'adc_vb_f': '12.549', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911284405 ms)
2025-02-18 09:04:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911344420 ms)
2025-02-18 09:04:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911344420 ms)
2025-02-18 09:04:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:04:25.400592+00:00', 'runtime': '911340.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911344421 ms)
2025-02-18 09:04:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:03:25.400201+00:00', 'runtime': '911280.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911344422 ms)
2025-02-18 09:05:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911404419 ms)
2025-02-18 09:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911404419 ms)
2025-02-18 09:05:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:05:25.400151+00:00', 'runtime': '911400.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911404420 ms)
2025-02-18 09:05:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:04:25.400592+00:00', 'runtime': '911340.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.549', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.1', 'ahl_t': '1597.1', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911404421 ms)
2025-02-18 09:05:41.247 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+911419066 ms)
2025-02-18 09:05:41.247 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+911419066 ms)
2025-02-18 09:05:41.247 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 52672)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+911419067 ms)
2025-02-18 09:06:07.278 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+911445097 ms)
2025-02-18 09:06:07.278 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+911445097 ms)
2025-02-18 09:06:07.278 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+911445098 ms)
2025-02-18 09:06:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911464403 ms)
2025-02-18 09:06:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911464403 ms)
2025-02-18 09:06:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:06:25.400193+00:00', 'runtime': '911460.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.531', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.534', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911464404 ms)
2025-02-18 09:06:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:05:25.400151+00:00', 'runtime': '911400.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.537', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911464404 ms)
2025-02-18 09:07:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911524404 ms)
2025-02-18 09:07:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911524405 ms)
2025-02-18 09:07:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:07:25.400211+00:00', 'runtime': '911520.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.534', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.534', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911524406 ms)
2025-02-18 09:07:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:06:25.400193+00:00', 'runtime': '911460.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.531', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.534', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911524406 ms)
2025-02-18 09:08:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911584403 ms)
2025-02-18 09:08:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911584403 ms)
2025-02-18 09:08:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:08:25.400209+00:00', 'runtime': '911580.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911584404 ms)
2025-02-18 09:08:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:07:25.400211+00:00', 'runtime': '911520.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.534', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.534', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911584405 ms)
2025-02-18 09:09:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911644420 ms)
2025-02-18 09:09:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911644420 ms)
2025-02-18 09:09:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:09:25.400282+00:00', 'runtime': '911640.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911644422 ms)
2025-02-18 09:09:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:08:25.400209+00:00', 'runtime': '911580.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911644423 ms)
2025-02-18 09:09:35.699 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+911653518 ms)
2025-02-18 09:09:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+911653518 ms)
2025-02-18 09:09:35.701 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:09:25.400343+00:00'} (T+911653521 ms)
2025-02-18 09:10:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911704416 ms)
2025-02-18 09:10:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911704416 ms)
2025-02-18 09:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:10:25.400154+00:00', 'runtime': '911700.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911704417 ms)
2025-02-18 09:10:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:09:25.400282+00:00', 'runtime': '911640.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911704418 ms)
2025-02-18 09:11:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911764418 ms)
2025-02-18 09:11:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911764418 ms)
2025-02-18 09:11:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:11:25.400162+00:00', 'runtime': '911760.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911764419 ms)
2025-02-18 09:11:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:10:25.400154+00:00', 'runtime': '911700.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.543', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911764420 ms)
2025-02-18 09:12:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911824414 ms)
2025-02-18 09:12:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911824414 ms)
2025-02-18 09:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:12:25.400516+00:00', 'runtime': '911820.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911824415 ms)
2025-02-18 09:12:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:11:25.400162+00:00', 'runtime': '911760.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.546', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911824416 ms)
2025-02-18 09:13:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911884411 ms)
2025-02-18 09:13:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911884411 ms)
2025-02-18 09:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:13:25.400167+00:00', 'runtime': '911880.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911884412 ms)
2025-02-18 09:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:12:25.400516+00:00', 'runtime': '911820.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911884413 ms)
2025-02-18 09:14:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+911944412 ms)
2025-02-18 09:14:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+911944413 ms)
2025-02-18 09:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:14:25.400637+00:00', 'runtime': '911940.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.534', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911944414 ms)
2025-02-18 09:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:13:25.400167+00:00', 'runtime': '911880.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+911944414 ms)
2025-02-18 09:15:11.021 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+911988840 ms)
2025-02-18 09:15:11.021 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+911988841 ms)
2025-02-18 09:15:11.022 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53280)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+911988841 ms)
2025-02-18 09:15:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912004411 ms)
2025-02-18 09:15:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912004411 ms)
2025-02-18 09:15:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:15:25.403193+00:00', 'runtime': '912000.004', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912004413 ms)
2025-02-18 09:15:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:14:25.400637+00:00', 'runtime': '911940.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.534', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.546', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912004413 ms)
2025-02-18 09:16:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912064410 ms)
2025-02-18 09:16:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912064410 ms)
2025-02-18 09:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:16:25.400174+00:00', 'runtime': '912060.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912064411 ms)
2025-02-18 09:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:15:25.403193+00:00', 'runtime': '912000.004', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.531', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912064412 ms)
2025-02-18 09:17:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912124411 ms)
2025-02-18 09:17:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912124412 ms)
2025-02-18 09:17:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:17:25.400187+00:00', 'runtime': '912120.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.518', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912124413 ms)
2025-02-18 09:17:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:16:25.400174+00:00', 'runtime': '912060.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912124413 ms)
2025-02-18 09:17:32.289 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912130108 ms)
2025-02-18 09:17:32.289 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912130109 ms)
2025-02-18 09:17:32.290 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53438)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912130109 ms)
2025-02-18 09:18:20.420 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912178239 ms)
2025-02-18 09:18:20.420 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912178239 ms)
2025-02-18 09:18:20.421 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53490)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912178240 ms)
2025-02-18 09:18:23.498 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912181318 ms)
2025-02-18 09:18:23.499 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912181318 ms)
2025-02-18 09:18:23.499 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53494)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912181319 ms)
2025-02-18 09:18:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912184412 ms)
2025-02-18 09:18:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912184412 ms)
2025-02-18 09:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:18:25.400253+00:00', 'runtime': '912180.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912184416 ms)
2025-02-18 09:18:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:17:25.400187+00:00', 'runtime': '912120.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.518', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912184416 ms)
2025-02-18 09:18:35.728 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912193547 ms)
2025-02-18 09:18:35.728 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912193548 ms)
2025-02-18 09:18:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:18:25.400345+00:00'} (T+912193549 ms)
2025-02-18 09:19:15.564 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233383 ms)
2025-02-18 09:19:15.564 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233383 ms)
2025-02-18 09:19:15.565 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233384 ms)
2025-02-18 09:19:15.566 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233385 ms)
2025-02-18 09:19:15.566 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233385 ms)
2025-02-18 09:19:15.567 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233386 ms)
2025-02-18 09:19:15.568 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233387 ms)
2025-02-18 09:19:15.568 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233387 ms)
2025-02-18 09:19:15.568 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233388 ms)
2025-02-18 09:19:15.570 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233389 ms)
2025-02-18 09:19:15.570 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233389 ms)
2025-02-18 09:19:15.571 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233390 ms)
2025-02-18 09:19:15.572 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233391 ms)
2025-02-18 09:19:15.572 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233391 ms)
2025-02-18 09:19:15.572 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233392 ms)
2025-02-18 09:19:15.573 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233393 ms)
2025-02-18 09:19:15.574 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233393 ms)
2025-02-18 09:19:15.574 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233393 ms)
2025-02-18 09:19:15.575 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233394 ms)
2025-02-18 09:19:15.575 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233395 ms)
2025-02-18 09:19:15.576 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233395 ms)
2025-02-18 09:19:15.577 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233396 ms)
2025-02-18 09:19:15.577 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233396 ms)
2025-02-18 09:19:15.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233397 ms)
2025-02-18 09:19:15.578 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912233397 ms)
2025-02-18 09:19:15.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912233398 ms)
2025-02-18 09:19:15.579 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912233398 ms)
2025-02-18 09:19:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912244413 ms)
2025-02-18 09:19:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912244414 ms)
2025-02-18 09:19:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:19:25.400172+00:00', 'runtime': '912240.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.524', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912244415 ms)
2025-02-18 09:19:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:18:25.400253+00:00', 'runtime': '912180.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.2', 'ahl_t': '1597.2', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '3.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912244415 ms)
2025-02-18 09:20:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912304410 ms)
2025-02-18 09:20:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912304411 ms)
2025-02-18 09:20:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:20:25.400261+00:00', 'runtime': '912300.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.54', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912304411 ms)
2025-02-18 09:20:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:19:25.400172+00:00', 'runtime': '912240.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.524', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912304412 ms)
2025-02-18 09:21:24.827 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912362646 ms)
2025-02-18 09:21:24.827 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912362647 ms)
2025-02-18 09:21:24.828 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53686)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912362647 ms)
2025-02-18 09:21:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912364413 ms)
2025-02-18 09:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912364414 ms)
2025-02-18 09:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:21:25.400207+00:00', 'runtime': '912360.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.524', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912364415 ms)
2025-02-18 09:21:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:20:25.400261+00:00', 'runtime': '912300.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.54', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912364415 ms)
2025-02-18 09:22:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912424413 ms)
2025-02-18 09:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912424413 ms)
2025-02-18 09:22:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:22:25.400175+00:00', 'runtime': '912420.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912424414 ms)
2025-02-18 09:22:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:21:25.400207+00:00', 'runtime': '912360.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.524', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912424415 ms)
2025-02-18 09:23:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912484410 ms)
2025-02-18 09:23:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912484410 ms)
2025-02-18 09:23:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:23:25.400189+00:00', 'runtime': '912480.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.518', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912484411 ms)
2025-02-18 09:23:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:22:25.400175+00:00', 'runtime': '912420.001', 'adc_vb_f': '12.546', 'adc_va_f': '1.178', 'adc_vl_f': '12.537', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912484412 ms)
2025-02-18 09:24:25.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912543403 ms)
2025-02-18 09:24:25.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912543403 ms)
2025-02-18 09:24:25.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:24:25.400217+00:00', 'runtime': '912540.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912543404 ms)
2025-02-18 09:24:25.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:23:25.400189+00:00', 'runtime': '912480.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.518', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.543', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912543405 ms)
2025-02-18 09:24:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912553422 ms)
2025-02-18 09:24:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912553423 ms)
2025-02-18 09:24:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:24:25.400343+00:00'} (T+912553424 ms)
2025-02-18 09:25:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912604412 ms)
2025-02-18 09:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912604413 ms)
2025-02-18 09:25:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:25:25.400226+00:00', 'runtime': '912600.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912604414 ms)
2025-02-18 09:25:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:24:25.400217+00:00', 'runtime': '912540.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.524', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912604414 ms)
2025-02-18 09:25:35.596 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912613415 ms)
2025-02-18 09:25:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912613416 ms)
2025-02-18 09:25:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:25:25.400234+00:00'} (T+912613417 ms)
2025-02-18 09:25:49.165 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912626985 ms)
2025-02-18 09:25:49.166 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912626985 ms)
2025-02-18 09:25:49.166 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912626986 ms)
2025-02-18 09:25:49.168 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912626987 ms)
2025-02-18 09:25:49.168 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912626987 ms)
2025-02-18 09:25:49.169 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912626988 ms)
2025-02-18 09:25:49.170 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+912626990 ms)
2025-02-18 09:25:49.171 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+912626990 ms)
2025-02-18 09:25:49.173 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912626993 ms)
2025-02-18 09:26:08.297 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912646117 ms)
2025-02-18 09:26:08.298 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912646117 ms)
2025-02-18 09:26:08.298 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 53996)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912646117 ms)
2025-02-18 09:26:26.388 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912664208 ms)
2025-02-18 09:26:26.389 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912664208 ms)
2025-02-18 09:26:26.389 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54016)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912664209 ms)
2025-02-18 09:26:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912664413 ms)
2025-02-18 09:26:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912664414 ms)
2025-02-18 09:26:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:26:25.400175+00:00', 'runtime': '912660.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.518', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912664415 ms)
2025-02-18 09:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:25:25.400226+00:00', 'runtime': '912600.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912664415 ms)
2025-02-18 09:26:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912673419 ms)
2025-02-18 09:26:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+912673419 ms)
2025-02-18 09:26:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:26:25.413121+00:00'} (T+912673424 ms)
2025-02-18 09:26:41.498 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912679318 ms)
2025-02-18 09:26:41.499 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912679318 ms)
2025-02-18 09:26:41.499 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54034)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912679319 ms)
2025-02-18 09:27:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912724412 ms)
2025-02-18 09:27:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912724413 ms)
2025-02-18 09:27:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:27:25.402882+00:00', 'runtime': '912720.003', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.521', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912724414 ms)
2025-02-18 09:27:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:26:25.400175+00:00', 'runtime': '912660.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.518', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912724414 ms)
2025-02-18 09:27:26.639 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912724458 ms)
2025-02-18 09:27:26.639 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912724458 ms)
2025-02-18 09:27:26.639 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54082)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912724459 ms)
2025-02-18 09:28:09.786 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912767606 ms)
2025-02-18 09:28:09.787 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912767606 ms)
2025-02-18 09:28:09.787 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54130)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912767607 ms)
2025-02-18 09:28:11.855 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912769675 ms)
2025-02-18 09:28:11.856 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912769675 ms)
2025-02-18 09:28:11.856 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54134)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912769675 ms)
2025-02-18 09:28:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912784414 ms)
2025-02-18 09:28:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912784414 ms)
2025-02-18 09:28:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:28:25.400720+00:00', 'runtime': '912780.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912784416 ms)
2025-02-18 09:28:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:27:25.402882+00:00', 'runtime': '912720.003', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.521', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912784417 ms)
2025-02-18 09:29:00.098 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912817917 ms)
2025-02-18 09:29:00.098 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912817918 ms)
2025-02-18 09:29:00.099 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54186)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912817918 ms)
2025-02-18 09:29:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912844412 ms)
2025-02-18 09:29:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912844412 ms)
2025-02-18 09:29:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:29:25.400199+00:00', 'runtime': '912840.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912844414 ms)
2025-02-18 09:29:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:28:25.400720+00:00', 'runtime': '912780.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912844415 ms)
2025-02-18 09:30:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912904413 ms)
2025-02-18 09:30:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912904413 ms)
2025-02-18 09:30:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:30:25.400631+00:00', 'runtime': '912900.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912904415 ms)
2025-02-18 09:30:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:29:25.400199+00:00', 'runtime': '912840.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912904415 ms)
2025-02-18 09:31:09.338 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+912947157 ms)
2025-02-18 09:31:09.338 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+912947158 ms)
2025-02-18 09:31:09.339 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54322)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+912947158 ms)
2025-02-18 09:31:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+912964413 ms)
2025-02-18 09:31:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+912964413 ms)
2025-02-18 09:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:31:25.400202+00:00', 'runtime': '912960.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912964415 ms)
2025-02-18 09:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:30:25.400631+00:00', 'runtime': '912900.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+912964415 ms)
2025-02-18 09:32:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913024415 ms)
2025-02-18 09:32:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913024415 ms)
2025-02-18 09:32:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:32:25.400159+00:00', 'runtime': '913020.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.531', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913024416 ms)
2025-02-18 09:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:31:25.400202+00:00', 'runtime': '912960.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.515', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.3', 'ahl_t': '1597.3', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913024417 ms)
2025-02-18 09:32:34.528 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913032347 ms)
2025-02-18 09:32:34.528 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913032348 ms)
2025-02-18 09:32:34.529 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54412)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913032348 ms)
2025-02-18 09:33:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913084414 ms)
2025-02-18 09:33:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913084414 ms)
2025-02-18 09:33:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:33:25.400162+00:00', 'runtime': '913080.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913084415 ms)
2025-02-18 09:33:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:32:25.400159+00:00', 'runtime': '913020.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.531', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913084416 ms)
2025-02-18 09:34:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913144414 ms)
2025-02-18 09:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913144414 ms)
2025-02-18 09:34:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:34:25.400192+00:00', 'runtime': '913140.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913144415 ms)
2025-02-18 09:34:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:33:25.400162+00:00', 'runtime': '913080.001', 'adc_vb_f': '12.543', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913144415 ms)
2025-02-18 09:34:33.736 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913151555 ms)
2025-02-18 09:34:33.736 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913151555 ms)
2025-02-18 09:34:33.736 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54536)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913151556 ms)
2025-02-18 09:34:35.586 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913153405 ms)
2025-02-18 09:34:35.587 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913153406 ms)
2025-02-18 09:34:35.588 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:34:25.400339+00:00'} (T+913153407 ms)
2025-02-18 09:34:52.837 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913170656 ms)
2025-02-18 09:34:52.837 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913170656 ms)
2025-02-18 09:34:52.838 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54560)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913170657 ms)
2025-02-18 09:35:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913204413 ms)
2025-02-18 09:35:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913204413 ms)
2025-02-18 09:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:35:25.400178+00:00', 'runtime': '913200.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913204414 ms)
2025-02-18 09:35:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:34:25.400192+00:00', 'runtime': '913140.001', 'adc_vb_f': '12.54', 'adc_va_f': '1.178', 'adc_vl_f': '12.512', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913204415 ms)
2025-02-18 09:35:35.594 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913213413 ms)
2025-02-18 09:35:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913213414 ms)
2025-02-18 09:35:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:35:25.400231+00:00'} (T+913213419 ms)
2025-02-18 09:36:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913264414 ms)
2025-02-18 09:36:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913264414 ms)
2025-02-18 09:36:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:36:25.400170+00:00', 'runtime': '913260.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913264415 ms)
2025-02-18 09:36:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:35:25.400178+00:00', 'runtime': '913200.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913264416 ms)
2025-02-18 09:36:35.623 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913273442 ms)
2025-02-18 09:36:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913273443 ms)
2025-02-18 09:36:35.629 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:36:25.400194+00:00'} (T+913273448 ms)
2025-02-18 09:37:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913324415 ms)
2025-02-18 09:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913324416 ms)
2025-02-18 09:37:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:37:25.400177+00:00', 'runtime': '913320.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913324417 ms)
2025-02-18 09:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:36:25.400170+00:00', 'runtime': '913260.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913324417 ms)
2025-02-18 09:37:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913333431 ms)
2025-02-18 09:37:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913333432 ms)
2025-02-18 09:37:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:37:25.400169+00:00'} (T+913333436 ms)
2025-02-18 09:37:59.137 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913356956 ms)
2025-02-18 09:37:59.137 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913356956 ms)
2025-02-18 09:37:59.137 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54770)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913356957 ms)
2025-02-18 09:38:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913384412 ms)
2025-02-18 09:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913384413 ms)
2025-02-18 09:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:38:25.400212+00:00', 'runtime': '913380.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913384414 ms)
2025-02-18 09:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:37:25.400177+00:00', 'runtime': '913320.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913384414 ms)
2025-02-18 09:38:35.594 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913393414 ms)
2025-02-18 09:38:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913393414 ms)
2025-02-18 09:38:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:38:25.400203+00:00'} (T+913393416 ms)
2025-02-18 09:39:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913444412 ms)
2025-02-18 09:39:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913444412 ms)
2025-02-18 09:39:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:39:25.400209+00:00', 'runtime': '913440.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913444413 ms)
2025-02-18 09:39:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:38:25.400212+00:00', 'runtime': '913380.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.537', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913444414 ms)
2025-02-18 09:40:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913504413 ms)
2025-02-18 09:40:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913504414 ms)
2025-02-18 09:40:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:40:25.400175+00:00', 'runtime': '913500.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913504415 ms)
2025-02-18 09:40:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:39:25.400209+00:00', 'runtime': '913440.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913504415 ms)
2025-02-18 09:40:57.415 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913535234 ms)
2025-02-18 09:40:57.415 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913535234 ms)
2025-02-18 09:40:57.415 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 54960)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913535235 ms)
2025-02-18 09:41:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913564410 ms)
2025-02-18 09:41:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913564411 ms)
2025-02-18 09:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:41:25.400178+00:00', 'runtime': '913560.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913564412 ms)
2025-02-18 09:41:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:40:25.400175+00:00', 'runtime': '913500.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913564412 ms)
2025-02-18 09:41:35.689 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913573508 ms)
2025-02-18 09:41:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913573508 ms)
2025-02-18 09:41:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:41:25.400336+00:00'} (T+913573513 ms)
2025-02-18 09:42:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913624410 ms)
2025-02-18 09:42:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913624410 ms)
2025-02-18 09:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:42:25.400205+00:00', 'runtime': '913620.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913624411 ms)
2025-02-18 09:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:41:25.400178+00:00', 'runtime': '913560.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.534', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913624411 ms)
2025-02-18 09:42:55.615 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913653434 ms)
2025-02-18 09:42:55.615 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913653435 ms)
2025-02-18 09:42:55.616 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55088)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913653435 ms)
2025-02-18 09:42:58.713 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913656532 ms)
2025-02-18 09:42:58.713 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913656532 ms)
2025-02-18 09:42:58.714 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55092)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913656533 ms)
2025-02-18 09:43:12.815 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913670634 ms)
2025-02-18 09:43:12.815 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913670634 ms)
2025-02-18 09:43:12.815 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55108)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913670635 ms)
2025-02-18 09:43:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913684409 ms)
2025-02-18 09:43:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913684409 ms)
2025-02-18 09:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:43:25.400205+00:00', 'runtime': '913680.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913684410 ms)
2025-02-18 09:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:42:25.400205+00:00', 'runtime': '913620.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913684411 ms)
2025-02-18 09:43:27.935 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913685754 ms)
2025-02-18 09:43:27.935 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913685755 ms)
2025-02-18 09:43:27.936 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55124)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913685755 ms)
2025-02-18 09:43:55.075 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913712894 ms)
2025-02-18 09:43:55.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913712894 ms)
2025-02-18 09:43:55.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55154)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913712894 ms)
2025-02-18 09:44:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913744410 ms)
2025-02-18 09:44:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913744410 ms)
2025-02-18 09:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:44:25.400189+00:00', 'runtime': '913740.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913744411 ms)
2025-02-18 09:44:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:43:25.400205+00:00', 'runtime': '913680.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913744412 ms)
2025-02-18 09:44:40.217 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913758036 ms)
2025-02-18 09:44:40.217 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913758037 ms)
2025-02-18 09:44:40.218 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55202)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913758037 ms)
2025-02-18 09:45:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913804409 ms)
2025-02-18 09:45:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913804410 ms)
2025-02-18 09:45:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:45:25.400196+00:00', 'runtime': '913800.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913804411 ms)
2025-02-18 09:45:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:44:25.400189+00:00', 'runtime': '913740.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913804411 ms)
2025-02-18 09:46:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913864412 ms)
2025-02-18 09:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913864413 ms)
2025-02-18 09:46:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:46:25.400174+00:00', 'runtime': '913860.001', 'adc_vb_f': '12.527', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913864414 ms)
2025-02-18 09:46:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:45:25.400196+00:00', 'runtime': '913800.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.4', 'ahl_t': '1597.4', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913864414 ms)
2025-02-18 09:47:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913924411 ms)
2025-02-18 09:47:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913924412 ms)
2025-02-18 09:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:47:25.400173+00:00', 'runtime': '913920.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913924416 ms)
2025-02-18 09:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:46:25.400174+00:00', 'runtime': '913860.001', 'adc_vb_f': '12.527', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913924416 ms)
2025-02-18 09:47:35.593 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913933412 ms)
2025-02-18 09:47:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+913933413 ms)
2025-02-18 09:47:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:47:25.400327+00:00'} (T+913933414 ms)
2025-02-18 09:48:09.583 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913967402 ms)
2025-02-18 09:48:09.583 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913967403 ms)
2025-02-18 09:48:09.584 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55432)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913967403 ms)
2025-02-18 09:48:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+913984413 ms)
2025-02-18 09:48:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+913984413 ms)
2025-02-18 09:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:48:25.400159+00:00', 'runtime': '913980.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913984417 ms)
2025-02-18 09:48:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:47:25.400173+00:00', 'runtime': '913920.001', 'adc_vb_f': '12.534', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.531', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+913984417 ms)
2025-02-18 09:48:31.664 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+913989483 ms)
2025-02-18 09:48:31.664 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+913989483 ms)
2025-02-18 09:48:31.664 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55458)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+913989484 ms)
2025-02-18 09:49:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914044417 ms)
2025-02-18 09:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914044417 ms)
2025-02-18 09:49:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:49:25.400193+00:00', 'runtime': '914040.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914044419 ms)
2025-02-18 09:49:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:48:25.400159+00:00', 'runtime': '913980.001', 'adc_vb_f': '12.537', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914044419 ms)
2025-02-18 09:49:42.844 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+914060664 ms)
2025-02-18 09:49:42.845 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+914060664 ms)
2025-02-18 09:49:42.845 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55532)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+914060664 ms)
2025-02-18 09:50:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914104413 ms)
2025-02-18 09:50:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914104414 ms)
2025-02-18 09:50:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:50:25.400187+00:00', 'runtime': '914100.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914104415 ms)
2025-02-18 09:50:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:49:25.400193+00:00', 'runtime': '914040.001', 'adc_vb_f': '12.531', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914104415 ms)
2025-02-18 09:50:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914113429 ms)
2025-02-18 09:50:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914113429 ms)
2025-02-18 09:50:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:50:25.400336+00:00'} (T+914113434 ms)
2025-02-18 09:51:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914164414 ms)
2025-02-18 09:51:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914164415 ms)
2025-02-18 09:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:51:25.400182+00:00', 'runtime': '914160.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914164419 ms)
2025-02-18 09:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:50:25.400187+00:00', 'runtime': '914100.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914164420 ms)
2025-02-18 09:51:35.591 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914173410 ms)
2025-02-18 09:51:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914173410 ms)
2025-02-18 09:51:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:51:25.400185+00:00'} (T+914173412 ms)
2025-02-18 09:52:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914224416 ms)
2025-02-18 09:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914224416 ms)
2025-02-18 09:52:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:52:25.400153+00:00', 'runtime': '914220.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914224421 ms)
2025-02-18 09:52:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:51:25.400182+00:00', 'runtime': '914160.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914224421 ms)
2025-02-18 09:53:16.192 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+914274012 ms)
2025-02-18 09:53:16.193 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+914274012 ms)
2025-02-18 09:53:16.193 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 55760)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+914274013 ms)
2025-02-18 09:53:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914284415 ms)
2025-02-18 09:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914284415 ms)
2025-02-18 09:53:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:53:25.400196+00:00', 'runtime': '914280.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914284419 ms)
2025-02-18 09:53:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:52:25.400153+00:00', 'runtime': '914220.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914284420 ms)
2025-02-18 09:54:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914344414 ms)
2025-02-18 09:54:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914344415 ms)
2025-02-18 09:54:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:54:25.400160+00:00', 'runtime': '914340.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914344416 ms)
2025-02-18 09:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:53:25.400196+00:00', 'runtime': '914280.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20140', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914344416 ms)
2025-02-18 09:54:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914353427 ms)
2025-02-18 09:54:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914353428 ms)
2025-02-18 09:54:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:54:25.400335+00:00'} (T+914353429 ms)
2025-02-18 09:55:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914404414 ms)
2025-02-18 09:55:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914404414 ms)
2025-02-18 09:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:55:25.400205+00:00', 'runtime': '914400.001', 'adc_vb_f': '12.527', 'adc_va_f': '1.184', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914404415 ms)
2025-02-18 09:55:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:54:25.400160+00:00', 'runtime': '914340.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914404415 ms)
2025-02-18 09:56:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914464415 ms)
2025-02-18 09:56:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914464416 ms)
2025-02-18 09:56:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:56:25.400203+00:00', 'runtime': '914460.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914464417 ms)
2025-02-18 09:56:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:55:25.400205+00:00', 'runtime': '914400.001', 'adc_vb_f': '12.527', 'adc_va_f': '1.184', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914464417 ms)
2025-02-18 09:57:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914524414 ms)
2025-02-18 09:57:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914524414 ms)
2025-02-18 09:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:57:25.400163+00:00', 'runtime': '914520.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914524415 ms)
2025-02-18 09:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:56:25.400203+00:00', 'runtime': '914460.001', 'adc_vb_f': '12.524', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914524415 ms)
2025-02-18 09:57:47.612 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+914545431 ms)
2025-02-18 09:57:47.612 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+914545431 ms)
2025-02-18 09:57:47.613 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56056)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+914545432 ms)
2025-02-18 09:58:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914584414 ms)
2025-02-18 09:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914584414 ms)
2025-02-18 09:58:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:58:25.400160+00:00', 'runtime': '914580.001', 'adc_vb_f': '12.527', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914584415 ms)
2025-02-18 09:58:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:57:25.400163+00:00', 'runtime': '914520.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914584415 ms)
2025-02-18 09:59:14.813 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+914632633 ms)
2025-02-18 09:59:14.814 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+914632633 ms)
2025-02-18 09:59:14.814 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56148)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+914632634 ms)
2025-02-18 09:59:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914644410 ms)
2025-02-18 09:59:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914644411 ms)
2025-02-18 09:59:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 09:59:25.400218+00:00', 'runtime': '914640.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914644412 ms)
2025-02-18 09:59:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:58:25.400160+00:00', 'runtime': '914580.001', 'adc_vb_f': '12.527', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.5', 'ahl_t': '1597.5', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914644412 ms)
2025-02-18 09:59:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914653419 ms)
2025-02-18 09:59:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914653419 ms)
2025-02-18 09:59:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 09:59:25.400344+00:00'} (T+914653424 ms)
2025-02-18 10:00:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914704411 ms)
2025-02-18 10:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914704411 ms)
2025-02-18 10:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:00:25.400200+00:00', 'runtime': '914700.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914704412 ms)
2025-02-18 10:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 09:59:25.400218+00:00', 'runtime': '914640.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914704413 ms)
2025-02-18 10:01:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914764409 ms)
2025-02-18 10:01:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914764410 ms)
2025-02-18 10:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:01:25.400211+00:00', 'runtime': '914760.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914764411 ms)
2025-02-18 10:01:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:00:25.400200+00:00', 'runtime': '914700.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.524', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914764411 ms)
2025-02-18 10:02:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914824412 ms)
2025-02-18 10:02:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914824412 ms)
2025-02-18 10:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:02:25.400267+00:00', 'runtime': '914820.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914824413 ms)
2025-02-18 10:02:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:01:25.400211+00:00', 'runtime': '914760.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914824414 ms)
2025-02-18 10:03:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914884414 ms)
2025-02-18 10:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914884414 ms)
2025-02-18 10:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:03:25.400264+00:00', 'runtime': '914880.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914884415 ms)
2025-02-18 10:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:02:25.400267+00:00', 'runtime': '914820.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914884416 ms)
2025-02-18 10:03:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914893421 ms)
2025-02-18 10:03:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914893422 ms)
2025-02-18 10:03:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:03:25.400337+00:00'} (T+914893429 ms)
2025-02-18 10:04:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+914944416 ms)
2025-02-18 10:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+914944416 ms)
2025-02-18 10:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:04:25.400198+00:00', 'runtime': '914940.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.184', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914944417 ms)
2025-02-18 10:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:03:25.400264+00:00', 'runtime': '914880.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+914944417 ms)
2025-02-18 10:04:35.258 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+914953077 ms)
2025-02-18 10:04:35.258 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+914953077 ms)
2025-02-18 10:04:35.258 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56488)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+914953078 ms)
2025-02-18 10:04:35.597 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914953416 ms)
2025-02-18 10:04:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+914953417 ms)
2025-02-18 10:04:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:04:25.400223+00:00'} (T+914953421 ms)
2025-02-18 10:04:52.357 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+914970176 ms)
2025-02-18 10:04:52.357 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+914970176 ms)
2025-02-18 10:04:52.358 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56506)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+914970177 ms)
2025-02-18 10:05:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915004418 ms)
2025-02-18 10:05:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915004418 ms)
2025-02-18 10:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:05:25.400200+00:00', 'runtime': '915000.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915004419 ms)
2025-02-18 10:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:04:25.400198+00:00', 'runtime': '914940.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.184', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.493', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915004420 ms)
2025-02-18 10:06:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915064416 ms)
2025-02-18 10:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915064416 ms)
2025-02-18 10:06:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:06:25.399997+00:00', 'runtime': '915060.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915064417 ms)
2025-02-18 10:06:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:05:25.400200+00:00', 'runtime': '915000.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915064418 ms)
2025-02-18 10:06:37.547 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915075367 ms)
2025-02-18 10:06:37.548 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915075367 ms)
2025-02-18 10:06:37.548 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56628)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915075367 ms)
2025-02-18 10:07:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915124416 ms)
2025-02-18 10:07:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915124417 ms)
2025-02-18 10:07:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:07:25.400577+00:00', 'runtime': '915120.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915124418 ms)
2025-02-18 10:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:06:25.399997+00:00', 'runtime': '915060.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915124419 ms)
2025-02-18 10:08:17.747 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915175566 ms)
2025-02-18 10:08:17.747 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915175566 ms)
2025-02-18 10:08:17.748 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56734)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915175567 ms)
2025-02-18 10:08:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915184413 ms)
2025-02-18 10:08:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915184414 ms)
2025-02-18 10:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:08:25.402786+00:00', 'runtime': '915180.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915184415 ms)
2025-02-18 10:08:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:07:25.400577+00:00', 'runtime': '915120.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915184415 ms)
2025-02-18 10:08:35.854 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915193674 ms)
2025-02-18 10:08:35.855 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915193674 ms)
2025-02-18 10:08:35.857 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:08:25.400159+00:00'} (T+915193676 ms)
2025-02-18 10:09:04.897 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915222717 ms)
2025-02-18 10:09:04.898 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915222717 ms)
2025-02-18 10:09:04.898 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56786)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915222718 ms)
2025-02-18 10:09:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915244412 ms)
2025-02-18 10:09:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915244412 ms)
2025-02-18 10:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:09:25.402806+00:00', 'runtime': '915240.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915244413 ms)
2025-02-18 10:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:08:25.402786+00:00', 'runtime': '915180.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915244414 ms)
2025-02-18 10:09:33.060 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915250879 ms)
2025-02-18 10:09:33.060 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915250879 ms)
2025-02-18 10:09:33.061 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56816)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915250880 ms)
2025-02-18 10:09:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915253420 ms)
2025-02-18 10:09:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915253421 ms)
2025-02-18 10:09:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:09:25.400233+00:00'} (T+915253423 ms)
2025-02-18 10:10:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915304410 ms)
2025-02-18 10:10:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915304411 ms)
2025-02-18 10:10:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:10:25.402784+00:00', 'runtime': '915300.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915304412 ms)
2025-02-18 10:10:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:09:25.402806+00:00', 'runtime': '915240.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915304412 ms)
2025-02-18 10:11:23.270 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915361089 ms)
2025-02-18 10:11:23.270 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915361089 ms)
2025-02-18 10:11:23.270 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56936)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915361090 ms)
2025-02-18 10:11:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915364411 ms)
2025-02-18 10:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915364411 ms)
2025-02-18 10:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:11:25.400741+00:00', 'runtime': '915360.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915364412 ms)
2025-02-18 10:11:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:10:25.402784+00:00', 'runtime': '915300.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915364413 ms)
2025-02-18 10:11:58.379 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915396198 ms)
2025-02-18 10:11:58.379 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915396199 ms)
2025-02-18 10:11:58.380 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56974)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915396199 ms)
2025-02-18 10:12:11.492 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915409311 ms)
2025-02-18 10:12:11.492 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915409312 ms)
2025-02-18 10:12:11.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 56988)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915409312 ms)
2025-02-18 10:12:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915424410 ms)
2025-02-18 10:12:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915424410 ms)
2025-02-18 10:12:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:12:25.400570+00:00', 'runtime': '915420.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915424411 ms)
2025-02-18 10:12:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:11:25.400741+00:00', 'runtime': '915360.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915424412 ms)
2025-02-18 10:13:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915484411 ms)
2025-02-18 10:13:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915484412 ms)
2025-02-18 10:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:13:25.402729+00:00', 'runtime': '915480.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915484413 ms)
2025-02-18 10:13:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:12:25.400570+00:00', 'runtime': '915420.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.6', 'ahl_t': '1597.6', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915484413 ms)
2025-02-18 10:13:35.786 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915493605 ms)
2025-02-18 10:13:35.786 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915493606 ms)
2025-02-18 10:13:35.787 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:13:25.400125+00:00'} (T+915493606 ms)
2025-02-18 10:13:55.780 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915513599 ms)
2025-02-18 10:13:55.780 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915513599 ms)
2025-02-18 10:13:55.781 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57100)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915513600 ms)
2025-02-18 10:13:58.880 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915516699 ms)
2025-02-18 10:13:58.880 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915516699 ms)
2025-02-18 10:13:58.881 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57104)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915516700 ms)
2025-02-18 10:14:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915544411 ms)
2025-02-18 10:14:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915544412 ms)
2025-02-18 10:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:14:25.402601+00:00', 'runtime': '915540.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915544413 ms)
2025-02-18 10:14:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:13:25.402729+00:00', 'runtime': '915480.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915544413 ms)
2025-02-18 10:14:31.990 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915549809 ms)
2025-02-18 10:14:31.990 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915549809 ms)
2025-02-18 10:14:31.990 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57138)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915549810 ms)
2025-02-18 10:14:35.587 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915553406 ms)
2025-02-18 10:14:35.588 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915553407 ms)
2025-02-18 10:14:35.589 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:14:25.400025+00:00'} (T+915553408 ms)
2025-02-18 10:15:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915604413 ms)
2025-02-18 10:15:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915604413 ms)
2025-02-18 10:15:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:15:25.400261+00:00', 'runtime': '915600.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915604414 ms)
2025-02-18 10:15:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:14:25.402601+00:00', 'runtime': '915540.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915604415 ms)
2025-02-18 10:15:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915613426 ms)
2025-02-18 10:15:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915613427 ms)
2025-02-18 10:15:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:15:25.401143+00:00'} (T+915613428 ms)
2025-02-18 10:16:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915664412 ms)
2025-02-18 10:16:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915664413 ms)
2025-02-18 10:16:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:16:25.400177+00:00', 'runtime': '915660.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915664414 ms)
2025-02-18 10:16:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:15:25.400261+00:00', 'runtime': '915600.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915664414 ms)
2025-02-18 10:16:53.247 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915691066 ms)
2025-02-18 10:16:53.247 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915691067 ms)
2025-02-18 10:16:53.248 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57302)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915691067 ms)
2025-02-18 10:17:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915724414 ms)
2025-02-18 10:17:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915724414 ms)
2025-02-18 10:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:17:25.400770+00:00', 'runtime': '915720.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915724416 ms)
2025-02-18 10:17:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:16:25.400177+00:00', 'runtime': '915660.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915724417 ms)
2025-02-18 10:18:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915784413 ms)
2025-02-18 10:18:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915784414 ms)
2025-02-18 10:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:18:25.400433+00:00', 'runtime': '915780.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915784415 ms)
2025-02-18 10:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:17:25.400770+00:00', 'runtime': '915720.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915784415 ms)
2025-02-18 10:19:16.663 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915834482 ms)
2025-02-18 10:19:16.663 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915834483 ms)
2025-02-18 10:19:16.664 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57450)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915834483 ms)
2025-02-18 10:19:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915844411 ms)
2025-02-18 10:19:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915844411 ms)
2025-02-18 10:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:19:25.400154+00:00', 'runtime': '915840.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915844412 ms)
2025-02-18 10:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:18:25.400433+00:00', 'runtime': '915780.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915844413 ms)
2025-02-18 10:19:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915853419 ms)
2025-02-18 10:19:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+915853420 ms)
2025-02-18 10:19:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:19:25.400339+00:00'} (T+915853425 ms)
2025-02-18 10:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915904412 ms)
2025-02-18 10:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915904413 ms)
2025-02-18 10:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:20:25.400233+00:00', 'runtime': '915900.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915904414 ms)
2025-02-18 10:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:19:25.400154+00:00', 'runtime': '915840.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915904414 ms)
2025-02-18 10:21:03.908 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+915941727 ms)
2025-02-18 10:21:03.908 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+915941727 ms)
2025-02-18 10:21:03.909 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57566)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+915941728 ms)
2025-02-18 10:21:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+915964412 ms)
2025-02-18 10:21:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+915964412 ms)
2025-02-18 10:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:21:25.400211+00:00', 'runtime': '915960.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915964413 ms)
2025-02-18 10:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:20:25.400233+00:00', 'runtime': '915900.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+915964414 ms)
2025-02-18 10:22:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916024409 ms)
2025-02-18 10:22:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916024410 ms)
2025-02-18 10:22:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:22:25.400179+00:00', 'runtime': '916020.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916024411 ms)
2025-02-18 10:22:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:21:25.400211+00:00', 'runtime': '915960.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916024411 ms)
2025-02-18 10:22:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916033427 ms)
2025-02-18 10:22:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916033427 ms)
2025-02-18 10:22:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:22:25.400331+00:00'} (T+916033429 ms)
2025-02-18 10:23:26.170 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916083989 ms)
2025-02-18 10:23:26.170 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916083990 ms)
2025-02-18 10:23:26.171 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57716)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916083990 ms)
2025-02-18 10:23:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916084409 ms)
2025-02-18 10:23:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916084410 ms)
2025-02-18 10:23:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:23:25.400177+00:00', 'runtime': '916080.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916084411 ms)
2025-02-18 10:23:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:22:25.400179+00:00', 'runtime': '916020.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916084411 ms)
2025-02-18 10:23:35.634 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916093453 ms)
2025-02-18 10:23:35.635 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916093454 ms)
2025-02-18 10:23:35.641 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:23:25.400176+00:00'} (T+916093460 ms)
2025-02-18 10:23:59.289 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916117108 ms)
2025-02-18 10:23:59.289 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916117109 ms)
2025-02-18 10:23:59.290 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57754)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916117109 ms)
2025-02-18 10:24:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916144409 ms)
2025-02-18 10:24:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916144409 ms)
2025-02-18 10:24:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:24:25.400269+00:00', 'runtime': '916140.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916144410 ms)
2025-02-18 10:24:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:23:25.400177+00:00', 'runtime': '916080.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916144411 ms)
2025-02-18 10:24:35.781 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916153601 ms)
2025-02-18 10:24:35.782 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916153601 ms)
2025-02-18 10:24:35.783 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:24:25.400262+00:00'} (T+916153603 ms)
2025-02-18 10:25:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916204407 ms)
2025-02-18 10:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916204408 ms)
2025-02-18 10:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:25:25.400208+00:00', 'runtime': '916200.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916204412 ms)
2025-02-18 10:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:24:25.400269+00:00', 'runtime': '916140.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916204413 ms)
2025-02-18 10:26:11.511 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916249330 ms)
2025-02-18 10:26:11.511 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916249331 ms)
2025-02-18 10:26:11.512 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57906)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916249331 ms)
2025-02-18 10:26:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916264405 ms)
2025-02-18 10:26:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916264405 ms)
2025-02-18 10:26:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:26:25.400195+00:00', 'runtime': '916260.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916264408 ms)
2025-02-18 10:26:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:25:25.400208+00:00', 'runtime': '916200.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.7', 'ahl_t': '1597.7', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916264409 ms)
2025-02-18 10:26:27.608 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916265427 ms)
2025-02-18 10:26:27.608 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916265427 ms)
2025-02-18 10:26:27.609 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 57924)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916265428 ms)
2025-02-18 10:27:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916324420 ms)
2025-02-18 10:27:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916324420 ms)
2025-02-18 10:27:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:27:25.400212+00:00', 'runtime': '916320.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916324424 ms)
2025-02-18 10:27:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:26:25.400195+00:00', 'runtime': '916260.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916324425 ms)
2025-02-18 10:27:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916333437 ms)
2025-02-18 10:27:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916333437 ms)
2025-02-18 10:27:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:27:25.413056+00:00'} (T+916333443 ms)
2025-02-18 10:27:38.797 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916336616 ms)
2025-02-18 10:27:38.797 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916336617 ms)
2025-02-18 10:27:38.798 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58000)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916336617 ms)
2025-02-18 10:28:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916384419 ms)
2025-02-18 10:28:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916384420 ms)
2025-02-18 10:28:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:28:25.402772+00:00', 'runtime': '916380.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916384424 ms)
2025-02-18 10:28:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:27:25.400212+00:00', 'runtime': '916320.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916384425 ms)
2025-02-18 10:29:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916444406 ms)
2025-02-18 10:29:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916444406 ms)
2025-02-18 10:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:29:25.400042+00:00', 'runtime': '916440.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916444407 ms)
2025-02-18 10:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:28:25.402772+00:00', 'runtime': '916380.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916444410 ms)
2025-02-18 10:29:36.030 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916453849 ms)
2025-02-18 10:29:36.030 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916453850 ms)
2025-02-18 10:29:36.031 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58124)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916453850 ms)
2025-02-18 10:30:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916504402 ms)
2025-02-18 10:30:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916504403 ms)
2025-02-18 10:30:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:30:25.400584+00:00', 'runtime': '916500.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916504404 ms)
2025-02-18 10:30:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:29:25.400042+00:00', 'runtime': '916440.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916504404 ms)
2025-02-18 10:30:35.180 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916512999 ms)
2025-02-18 10:30:35.180 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916513000 ms)
2025-02-18 10:30:35.181 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58186)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916513000 ms)
2025-02-18 10:31:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916564420 ms)
2025-02-18 10:31:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916564420 ms)
2025-02-18 10:31:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:31:25.400268+00:00', 'runtime': '916560.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916564422 ms)
2025-02-18 10:31:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:30:25.400584+00:00', 'runtime': '916500.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916564422 ms)
2025-02-18 10:31:35.854 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916573673 ms)
2025-02-18 10:31:35.855 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916573674 ms)
2025-02-18 10:31:35.857 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:31:25.405064+00:00'} (T+916573676 ms)
2025-02-18 10:32:11.305 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+916609125 ms)
2025-02-18 10:32:11.306 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+916609125 ms)
2025-02-18 10:32:11.307 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916609126 ms)
2025-02-18 10:32:13.310 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+916611129 ms)
2025-02-18 10:32:13.310 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+916611129 ms)
2025-02-18 10:32:13.310 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916611129 ms)
2025-02-18 10:32:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916624417 ms)
2025-02-18 10:32:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916624418 ms)
2025-02-18 10:32:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:32:25.400185+00:00', 'runtime': '916620.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916624419 ms)
2025-02-18 10:32:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:31:25.400268+00:00', 'runtime': '916560.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916624419 ms)
2025-02-18 10:32:35.628 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916633447 ms)
2025-02-18 10:32:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916633448 ms)
2025-02-18 10:32:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:32:25.405133+00:00'} (T+916633450 ms)
2025-02-18 10:33:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916684417 ms)
2025-02-18 10:33:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916684417 ms)
2025-02-18 10:33:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:33:25.402752+00:00', 'runtime': '916680.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916684418 ms)
2025-02-18 10:33:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:32:25.400185+00:00', 'runtime': '916620.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916684419 ms)
2025-02-18 10:33:35.636 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916693455 ms)
2025-02-18 10:33:35.637 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916693456 ms)
2025-02-18 10:33:35.638 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:33:25.400171+00:00'} (T+916693457 ms)
2025-02-18 10:34:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916744417 ms)
2025-02-18 10:34:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916744417 ms)
2025-02-18 10:34:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:34:25.402816+00:00', 'runtime': '916740.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916744419 ms)
2025-02-18 10:34:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:33:25.402752+00:00', 'runtime': '916680.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916744419 ms)
2025-02-18 10:34:35.819 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916753638 ms)
2025-02-18 10:34:35.819 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916753639 ms)
2025-02-18 10:34:35.820 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:34:25.400248+00:00'} (T+916753640 ms)
2025-02-18 10:35:00.619 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916778438 ms)
2025-02-18 10:35:00.619 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916778439 ms)
2025-02-18 10:35:00.620 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58470)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916778439 ms)
2025-02-18 10:35:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916804418 ms)
2025-02-18 10:35:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916804419 ms)
2025-02-18 10:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:35:25.400156+00:00', 'runtime': '916800.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.002', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916804422 ms)
2025-02-18 10:35:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:34:25.402816+00:00', 'runtime': '916740.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916804422 ms)
2025-02-18 10:35:36.207 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916814026 ms)
2025-02-18 10:35:36.207 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+916814027 ms)
2025-02-18 10:35:36.209 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:35:25.400000+00:00'} (T+916814028 ms)
2025-02-18 10:36:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916864421 ms)
2025-02-18 10:36:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916864422 ms)
2025-02-18 10:36:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:36:25.400085+00:00', 'runtime': '916860.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916864423 ms)
2025-02-18 10:36:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:35:25.400156+00:00', 'runtime': '916800.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.002', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916864423 ms)
2025-02-18 10:37:11.874 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+916909693 ms)
2025-02-18 10:37:11.874 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+916909693 ms)
2025-02-18 10:37:11.874 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58620)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+916909694 ms)
2025-02-18 10:37:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916924415 ms)
2025-02-18 10:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916924415 ms)
2025-02-18 10:37:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:37:25.400173+00:00', 'runtime': '916920.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916924416 ms)
2025-02-18 10:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:36:25.400085+00:00', 'runtime': '916860.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.403', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916924417 ms)
2025-02-18 10:38:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+916984415 ms)
2025-02-18 10:38:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+916984415 ms)
2025-02-18 10:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:38:25.400201+00:00', 'runtime': '916980.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916984416 ms)
2025-02-18 10:38:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:37:25.400173+00:00', 'runtime': '916920.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+916984417 ms)
2025-02-18 10:39:19.133 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+917036953 ms)
2025-02-18 10:39:19.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+917036953 ms)
2025-02-18 10:39:19.134 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 58752)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917036953 ms)
2025-02-18 10:39:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917044413 ms)
2025-02-18 10:39:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917044413 ms)
2025-02-18 10:39:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:39:25.400262+00:00', 'runtime': '917040.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917044414 ms)
2025-02-18 10:39:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:38:25.400201+00:00', 'runtime': '916980.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917044415 ms)
2025-02-18 10:40:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917104415 ms)
2025-02-18 10:40:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917104415 ms)
2025-02-18 10:40:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:40:25.400172+00:00', 'runtime': '917100.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917104416 ms)
2025-02-18 10:40:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:39:25.400262+00:00', 'runtime': '917040.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.8', 'ahl_t': '1597.8', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917104417 ms)
2025-02-18 10:41:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917164415 ms)
2025-02-18 10:41:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917164416 ms)
2025-02-18 10:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:41:25.400221+00:00', 'runtime': '917160.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917164417 ms)
2025-02-18 10:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:40:25.400172+00:00', 'runtime': '917100.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917164417 ms)
2025-02-18 10:42:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917224416 ms)
2025-02-18 10:42:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917224417 ms)
2025-02-18 10:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:42:25.400259+00:00', 'runtime': '917220.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917224418 ms)
2025-02-18 10:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:41:25.400221+00:00', 'runtime': '917160.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917224418 ms)
2025-02-18 10:43:14.544 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+917272363 ms)
2025-02-18 10:43:14.544 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+917272363 ms)
2025-02-18 10:43:14.544 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59000)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917272364 ms)
2025-02-18 10:43:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917284414 ms)
2025-02-18 10:43:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917284414 ms)
2025-02-18 10:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:43:25.400205+00:00', 'runtime': '917280.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917284415 ms)
2025-02-18 10:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:42:25.400259+00:00', 'runtime': '917220.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.418', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917284416 ms)
2025-02-18 10:44:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917344413 ms)
2025-02-18 10:44:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917344414 ms)
2025-02-18 10:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:44:25.400166+00:00', 'runtime': '917340.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917344415 ms)
2025-02-18 10:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:43:25.400205+00:00', 'runtime': '917280.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917344415 ms)
2025-02-18 10:45:04.680 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+917382500 ms)
2025-02-18 10:45:04.681 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+917382500 ms)
2025-02-18 10:45:04.681 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917382500 ms)
2025-02-18 10:45:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917404411 ms)
2025-02-18 10:45:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917404412 ms)
2025-02-18 10:45:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:45:25.400164+00:00', 'runtime': '917400.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917404413 ms)
2025-02-18 10:45:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:44:25.400166+00:00', 'runtime': '917340.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917404413 ms)
2025-02-18 10:46:25.690 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917463509 ms)
2025-02-18 10:46:25.690 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917463510 ms)
2025-02-18 10:46:25.694 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:46:25.400603+00:00', 'runtime': '917460.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917463514 ms)
2025-02-18 10:46:25.695 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:45:25.400164+00:00', 'runtime': '917400.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917463514 ms)
2025-02-18 10:47:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917524410 ms)
2025-02-18 10:47:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917524411 ms)
2025-02-18 10:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:47:25.402688+00:00', 'runtime': '917520.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917524415 ms)
2025-02-18 10:47:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:46:25.400603+00:00', 'runtime': '917460.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.389', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917524415 ms)
2025-02-18 10:47:49.957 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+917547776 ms)
2025-02-18 10:47:49.957 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+917547777 ms)
2025-02-18 10:47:49.958 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59300)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917547777 ms)
2025-02-18 10:48:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917584412 ms)
2025-02-18 10:48:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917584412 ms)
2025-02-18 10:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:48:25.400767+00:00', 'runtime': '917580.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917584417 ms)
2025-02-18 10:48:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:47:25.402688+00:00', 'runtime': '917520.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917584417 ms)
2025-02-18 10:48:36.087 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+917593906 ms)
2025-02-18 10:48:36.088 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+917593907 ms)
2025-02-18 10:48:36.088 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59348)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917593907 ms)
2025-02-18 10:49:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917644411 ms)
2025-02-18 10:49:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917644411 ms)
2025-02-18 10:49:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:49:25.400162+00:00', 'runtime': '917640.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917644413 ms)
2025-02-18 10:49:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:48:25.400767+00:00', 'runtime': '917580.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.521', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917644414 ms)
2025-02-18 10:50:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917704412 ms)
2025-02-18 10:50:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917704413 ms)
2025-02-18 10:50:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:50:25.400678+00:00', 'runtime': '917700.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917704415 ms)
2025-02-18 10:50:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:49:25.400162+00:00', 'runtime': '917640.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917704415 ms)
2025-02-18 10:51:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917764408 ms)
2025-02-18 10:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917764409 ms)
2025-02-18 10:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:51:25.400602+00:00', 'runtime': '917760.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917764411 ms)
2025-02-18 10:51:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:50:25.400678+00:00', 'runtime': '917700.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917764411 ms)
2025-02-18 10:51:29.401 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+917767220 ms)
2025-02-18 10:51:29.401 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+917767220 ms)
2025-02-18 10:51:29.401 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59530)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917767221 ms)
2025-02-18 10:52:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917824422 ms)
2025-02-18 10:52:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917824422 ms)
2025-02-18 10:52:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:52:25.400202+00:00', 'runtime': '917820.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917824423 ms)
2025-02-18 10:52:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:51:25.400602+00:00', 'runtime': '917760.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917824426 ms)
2025-02-18 10:52:35.705 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+917833524 ms)
2025-02-18 10:52:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+917833525 ms)
2025-02-18 10:52:35.707 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 10:52:25.413052+00:00'} (T+917833526 ms)
2025-02-18 10:52:53.578 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+917851397 ms)
2025-02-18 10:52:53.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+917851397 ms)
2025-02-18 10:52:53.578 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 59622)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917851398 ms)
2025-02-18 10:53:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917884408 ms)
2025-02-18 10:53:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917884408 ms)
2025-02-18 10:53:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:53:25.402778+00:00', 'runtime': '917880.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917884409 ms)
2025-02-18 10:53:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:52:25.400202+00:00', 'runtime': '917820.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917884410 ms)
2025-02-18 10:54:15.680 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+917933500 ms)
2025-02-18 10:54:15.681 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+917933500 ms)
2025-02-18 10:54:15.681 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+917933500 ms)
2025-02-18 10:54:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+917944407 ms)
2025-02-18 10:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+917944407 ms)
2025-02-18 10:54:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:54:25.400769+00:00', 'runtime': '917940.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917944409 ms)
2025-02-18 10:54:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:53:25.402778+00:00', 'runtime': '917880.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1597.9', 'ahl_t': '1597.9', 'hourmeter': '20141', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+917944410 ms)
2025-02-18 10:55:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918004407 ms)
2025-02-18 10:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918004407 ms)
2025-02-18 10:55:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:55:25.400160+00:00', 'runtime': '918000.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918004409 ms)
2025-02-18 10:55:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:54:25.400769+00:00', 'runtime': '917940.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918004409 ms)
2025-02-18 10:56:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918064406 ms)
2025-02-18 10:56:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918064407 ms)
2025-02-18 10:56:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:56:25.400281+00:00', 'runtime': '918060.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918064409 ms)
2025-02-18 10:56:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:55:25.400160+00:00', 'runtime': '918000.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.518', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918064409 ms)
2025-02-18 10:57:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918124406 ms)
2025-02-18 10:57:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918124406 ms)
2025-02-18 10:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:57:25.400179+00:00', 'runtime': '918120.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918124411 ms)
2025-02-18 10:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:56:25.400281+00:00', 'runtime': '918060.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918124411 ms)
2025-02-18 10:58:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918184403 ms)
2025-02-18 10:58:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918184404 ms)
2025-02-18 10:58:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:58:25.400175+00:00', 'runtime': '918180.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918184408 ms)
2025-02-18 10:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:57:25.400179+00:00', 'runtime': '918120.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918184408 ms)
2025-02-18 10:59:06.108 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918223927 ms)
2025-02-18 10:59:06.108 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918223927 ms)
2025-02-18 10:59:06.108 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60024)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918223927 ms)
2025-02-18 10:59:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918244403 ms)
2025-02-18 10:59:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918244403 ms)
2025-02-18 10:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 10:59:25.400604+00:00', 'runtime': '918240.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918244406 ms)
2025-02-18 10:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:58:25.400175+00:00', 'runtime': '918180.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918244407 ms)
2025-02-18 11:00:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918304418 ms)
2025-02-18 11:00:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918304419 ms)
2025-02-18 11:00:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:00:25.402760+00:00', 'runtime': '918300.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918304423 ms)
2025-02-18 11:00:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 10:59:25.400604+00:00', 'runtime': '918240.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918304423 ms)
2025-02-18 11:00:36.007 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918313826 ms)
2025-02-18 11:00:36.008 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918313827 ms)
2025-02-18 11:00:36.009 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:00:25.400156+00:00'} (T+918313828 ms)
2025-02-18 11:01:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918364415 ms)
2025-02-18 11:01:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918364416 ms)
2025-02-18 11:01:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:01:25.402863+00:00', 'runtime': '918360.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.512', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918364417 ms)
2025-02-18 11:01:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:00:25.402760+00:00', 'runtime': '918300.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.515', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918364417 ms)
2025-02-18 11:01:35.685 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918373505 ms)
2025-02-18 11:01:35.686 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918373505 ms)
2025-02-18 11:01:35.687 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:01:25.400266+00:00'} (T+918373506 ms)
2025-02-18 11:02:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918424415 ms)
2025-02-18 11:02:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918424415 ms)
2025-02-18 11:02:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:02:25.400198+00:00', 'runtime': '918420.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918424416 ms)
2025-02-18 11:02:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:01:25.402863+00:00', 'runtime': '918360.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.512', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918424417 ms)
2025-02-18 11:03:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918484413 ms)
2025-02-18 11:03:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918484413 ms)
2025-02-18 11:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:03:25.400248+00:00', 'runtime': '918480.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918484414 ms)
2025-02-18 11:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:02:25.400198+00:00', 'runtime': '918420.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.449', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918484414 ms)
2025-02-18 11:03:37.527 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918495346 ms)
2025-02-18 11:03:37.527 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918495346 ms)
2025-02-18 11:03:37.528 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60310)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918495347 ms)
2025-02-18 11:04:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918544415 ms)
2025-02-18 11:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918544416 ms)
2025-02-18 11:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:04:25.400173+00:00', 'runtime': '918540.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918544417 ms)
2025-02-18 11:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:03:25.400248+00:00', 'runtime': '918480.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.432', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918544417 ms)
2025-02-18 11:04:35.596 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918553415 ms)
2025-02-18 11:04:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918553416 ms)
2025-02-18 11:04:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:04:25.400333+00:00'} (T+918553417 ms)
2025-02-18 11:05:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918604414 ms)
2025-02-18 11:05:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918604414 ms)
2025-02-18 11:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:05:25.400161+00:00', 'runtime': '918600.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918604415 ms)
2025-02-18 11:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:04:25.400173+00:00', 'runtime': '918540.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918604416 ms)
2025-02-18 11:06:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918664413 ms)
2025-02-18 11:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918664414 ms)
2025-02-18 11:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:06:25.400160+00:00', 'runtime': '918660.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918664415 ms)
2025-02-18 11:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:05:25.400161+00:00', 'runtime': '918600.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.411', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.509', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918664415 ms)
2025-02-18 11:07:22.917 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918720736 ms)
2025-02-18 11:07:22.917 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918720736 ms)
2025-02-18 11:07:22.918 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60558)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918720737 ms)
2025-02-18 11:07:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918724412 ms)
2025-02-18 11:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918724413 ms)
2025-02-18 11:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:07:25.400166+00:00', 'runtime': '918720.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918724414 ms)
2025-02-18 11:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:06:25.400160+00:00', 'runtime': '918660.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.391', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.0', 'ahl_t': '1598.0', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918724414 ms)
2025-02-18 11:07:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918733423 ms)
2025-02-18 11:07:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918733424 ms)
2025-02-18 11:07:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:07:25.400329+00:00'} (T+918733425 ms)
2025-02-18 11:07:44.017 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918741836 ms)
2025-02-18 11:07:44.017 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918741836 ms)
2025-02-18 11:07:44.018 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60582)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918741837 ms)
2025-02-18 11:08:12.145 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918769965 ms)
2025-02-18 11:08:12.146 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918769965 ms)
2025-02-18 11:08:12.146 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60614)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918769966 ms)
2025-02-18 11:08:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918784412 ms)
2025-02-18 11:08:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918784412 ms)
2025-02-18 11:08:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:08:25.400311+00:00', 'runtime': '918780.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918784413 ms)
2025-02-18 11:08:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:07:25.400166+00:00', 'runtime': '918720.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918784413 ms)
2025-02-18 11:08:41.259 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918799078 ms)
2025-02-18 11:08:41.259 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918799078 ms)
2025-02-18 11:08:41.260 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60646)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918799079 ms)
2025-02-18 11:09:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918844410 ms)
2025-02-18 11:09:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918844410 ms)
2025-02-18 11:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:09:25.400202+00:00', 'runtime': '918840.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918844411 ms)
2025-02-18 11:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:08:25.400311+00:00', 'runtime': '918780.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918844411 ms)
2025-02-18 11:09:45.387 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918863206 ms)
2025-02-18 11:09:45.387 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918863206 ms)
2025-02-18 11:09:45.387 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60714)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918863207 ms)
2025-02-18 11:10:25.687 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918903506 ms)
2025-02-18 11:10:25.687 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918903507 ms)
2025-02-18 11:10:25.688 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:10:25.400166+00:00', 'runtime': '918900.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918903508 ms)
2025-02-18 11:10:25.689 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:09:25.400202+00:00', 'runtime': '918840.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918903508 ms)
2025-02-18 11:10:35.832 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918913651 ms)
2025-02-18 11:10:35.833 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+918913652 ms)
2025-02-18 11:10:35.834 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:10:25.400319+00:00'} (T+918913653 ms)
2025-02-18 11:11:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+918964407 ms)
2025-02-18 11:11:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+918964408 ms)
2025-02-18 11:11:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:11:25.400072+00:00', 'runtime': '918960.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918964409 ms)
2025-02-18 11:11:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:10:25.400166+00:00', 'runtime': '918900.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+918964409 ms)
2025-02-18 11:11:31.567 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+918969386 ms)
2025-02-18 11:11:31.567 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+918969386 ms)
2025-02-18 11:11:31.568 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60830)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+918969387 ms)
2025-02-18 11:12:16.704 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919014524 ms)
2025-02-18 11:12:16.705 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919014524 ms)
2025-02-18 11:12:16.705 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60878)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919014524 ms)
2025-02-18 11:12:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919024406 ms)
2025-02-18 11:12:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919024407 ms)
2025-02-18 11:12:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:12:25.400179+00:00', 'runtime': '919020.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919024408 ms)
2025-02-18 11:12:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:11:25.400072+00:00', 'runtime': '918960.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.425', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919024408 ms)
2025-02-18 11:12:50.806 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919048626 ms)
2025-02-18 11:12:50.807 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919048626 ms)
2025-02-18 11:12:50.807 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 60914)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919048626 ms)
2025-02-18 11:13:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919084405 ms)
2025-02-18 11:13:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919084406 ms)
2025-02-18 11:13:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:13:25.400204+00:00', 'runtime': '919080.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919084407 ms)
2025-02-18 11:13:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:12:25.400179+00:00', 'runtime': '919020.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.396', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919084407 ms)
2025-02-18 11:13:35.693 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919093513 ms)
2025-02-18 11:13:35.694 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919093513 ms)
2025-02-18 11:13:35.695 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:13:25.400343+00:00'} (T+919093514 ms)
2025-02-18 11:14:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919144407 ms)
2025-02-18 11:14:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919144407 ms)
2025-02-18 11:14:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:14:25.400207+00:00', 'runtime': '919140.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919144408 ms)
2025-02-18 11:14:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:13:25.400204+00:00', 'runtime': '919080.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919144409 ms)
2025-02-18 11:14:35.583 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919153402 ms)
2025-02-18 11:14:35.584 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919153403 ms)
2025-02-18 11:14:35.585 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:14:25.400195+00:00'} (T+919153404 ms)
2025-02-18 11:15:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919204419 ms)
2025-02-18 11:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919204419 ms)
2025-02-18 11:15:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:15:25.400168+00:00', 'runtime': '919200.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919204420 ms)
2025-02-18 11:15:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:14:25.400207+00:00', 'runtime': '919140.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919204421 ms)
2025-02-18 11:15:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919213429 ms)
2025-02-18 11:15:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919213430 ms)
2025-02-18 11:15:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:15:25.400216+00:00'} (T+919213431 ms)
2025-02-18 11:15:42.104 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919219924 ms)
2025-02-18 11:15:42.105 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919219924 ms)
2025-02-18 11:15:42.105 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 32866)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919219925 ms)
2025-02-18 11:16:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919264415 ms)
2025-02-18 11:16:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919264416 ms)
2025-02-18 11:16:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:16:25.400162+00:00', 'runtime': '919260.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919264417 ms)
2025-02-18 11:16:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:15:25.400168+00:00', 'runtime': '919200.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919264417 ms)
2025-02-18 11:16:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919273419 ms)
2025-02-18 11:16:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919273420 ms)
2025-02-18 11:16:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:16:25.400182+00:00'} (T+919273421 ms)
2025-02-18 11:16:43.266 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919281085 ms)
2025-02-18 11:16:43.266 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919281086 ms)
2025-02-18 11:16:43.267 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 32942)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919281086 ms)
2025-02-18 11:17:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919324414 ms)
2025-02-18 11:17:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919324415 ms)
2025-02-18 11:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:17:25.400165+00:00', 'runtime': '919320.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919324416 ms)
2025-02-18 11:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:16:25.400162+00:00', 'runtime': '919260.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919324416 ms)
2025-02-18 11:17:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919333428 ms)
2025-02-18 11:17:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919333429 ms)
2025-02-18 11:17:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:17:25.400214+00:00'} (T+919333430 ms)
2025-02-18 11:18:09.454 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919367273 ms)
2025-02-18 11:18:09.454 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919367274 ms)
2025-02-18 11:18:09.455 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33036)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919367274 ms)
2025-02-18 11:18:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919384414 ms)
2025-02-18 11:18:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919384414 ms)
2025-02-18 11:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:18:25.400183+00:00', 'runtime': '919380.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919384415 ms)
2025-02-18 11:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:17:25.400165+00:00', 'runtime': '919320.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.454', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919384415 ms)
2025-02-18 11:18:36.210 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919394029 ms)
2025-02-18 11:18:36.210 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+919394030 ms)
2025-02-18 11:18:36.211 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:18:25.400206+00:00'} (T+919394031 ms)
2025-02-18 11:19:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919444413 ms)
2025-02-18 11:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919444413 ms)
2025-02-18 11:19:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:19:25.400210+00:00', 'runtime': '919440.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919444414 ms)
2025-02-18 11:19:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:18:25.400183+00:00', 'runtime': '919380.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919444415 ms)
2025-02-18 11:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919504413 ms)
2025-02-18 11:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919504413 ms)
2025-02-18 11:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:20:25.400840+00:00', 'runtime': '919500.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919504414 ms)
2025-02-18 11:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:19:25.400210+00:00', 'runtime': '919440.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919504415 ms)
2025-02-18 11:21:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919564412 ms)
2025-02-18 11:21:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919564413 ms)
2025-02-18 11:21:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:21:25.400596+00:00', 'runtime': '919560.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919564414 ms)
2025-02-18 11:21:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:20:25.400840+00:00', 'runtime': '919500.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.1', 'ahl_t': '1598.1', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919564414 ms)
2025-02-18 11:22:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919624412 ms)
2025-02-18 11:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919624412 ms)
2025-02-18 11:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:22:25.402744+00:00', 'runtime': '919620.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919624413 ms)
2025-02-18 11:22:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:21:25.400596+00:00', 'runtime': '919560.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919624414 ms)
2025-02-18 11:23:20.936 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919678755 ms)
2025-02-18 11:23:20.936 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919678756 ms)
2025-02-18 11:23:20.937 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33364)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919678756 ms)
2025-02-18 11:23:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919684409 ms)
2025-02-18 11:23:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919684410 ms)
2025-02-18 11:23:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:23:25.400807+00:00', 'runtime': '919680.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919684411 ms)
2025-02-18 11:23:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:22:25.402744+00:00', 'runtime': '919620.003', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919684411 ms)
2025-02-18 11:23:39.035 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919696854 ms)
2025-02-18 11:23:39.035 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919696854 ms)
2025-02-18 11:23:39.036 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33384)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919696855 ms)
2025-02-18 11:24:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919744422 ms)
2025-02-18 11:24:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919744423 ms)
2025-02-18 11:24:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:24:25.400537+00:00', 'runtime': '919740.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919744427 ms)
2025-02-18 11:24:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:23:25.400807+00:00', 'runtime': '919680.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.457', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919744427 ms)
2025-02-18 11:24:55.205 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919773025 ms)
2025-02-18 11:24:55.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919773025 ms)
2025-02-18 11:24:55.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33468)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919773025 ms)
2025-02-18 11:25:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919804405 ms)
2025-02-18 11:25:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919804406 ms)
2025-02-18 11:25:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:25:25.400022+00:00', 'runtime': '919800.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919804410 ms)
2025-02-18 11:25:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:24:25.400537+00:00', 'runtime': '919740.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919804410 ms)
2025-02-18 11:25:38.335 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919816154 ms)
2025-02-18 11:25:38.335 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919816154 ms)
2025-02-18 11:25:38.336 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33512)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919816155 ms)
2025-02-18 11:25:41.416 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919819235 ms)
2025-02-18 11:25:41.416 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919819236 ms)
2025-02-18 11:25:41.417 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33516)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919819236 ms)
2025-02-18 11:26:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919864405 ms)
2025-02-18 11:26:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919864406 ms)
2025-02-18 11:26:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:26:25.400608+00:00', 'runtime': '919860.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919864407 ms)
2025-02-18 11:26:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:25:25.400022+00:00', 'runtime': '919800.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919864407 ms)
2025-02-18 11:27:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919924422 ms)
2025-02-18 11:27:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919924422 ms)
2025-02-18 11:27:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:27:25.402805+00:00', 'runtime': '919920.003', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919924426 ms)
2025-02-18 11:27:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:26:25.400608+00:00', 'runtime': '919860.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919924426 ms)
2025-02-18 11:28:16.725 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919974544 ms)
2025-02-18 11:28:16.725 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919974544 ms)
2025-02-18 11:28:16.725 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33692)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919974545 ms)
2025-02-18 11:28:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+919984405 ms)
2025-02-18 11:28:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+919984406 ms)
2025-02-18 11:28:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:28:25.400650+00:00', 'runtime': '919980.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919984409 ms)
2025-02-18 11:28:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:27:25.402805+00:00', 'runtime': '919920.003', 'adc_vb_f': '12.518', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+919984409 ms)
2025-02-18 11:28:35.815 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+919993634 ms)
2025-02-18 11:28:35.815 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+919993635 ms)
2025-02-18 11:28:35.816 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33712)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+919993635 ms)
2025-02-18 11:29:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920044422 ms)
2025-02-18 11:29:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920044422 ms)
2025-02-18 11:29:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:29:25.400208+00:00', 'runtime': '920040.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920044426 ms)
2025-02-18 11:29:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:28:25.400650+00:00', 'runtime': '919980.001', 'adc_vb_f': '12.521', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.408', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920044426 ms)
2025-02-18 11:29:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920053421 ms)
2025-02-18 11:29:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920053422 ms)
2025-02-18 11:29:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:29:25.405064+00:00'} (T+920053423 ms)
2025-02-18 11:30:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920104420 ms)
2025-02-18 11:30:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920104420 ms)
2025-02-18 11:30:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:30:25.400174+00:00', 'runtime': '920100.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920104424 ms)
2025-02-18 11:30:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:29:25.400208+00:00', 'runtime': '920040.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920104425 ms)
2025-02-18 11:31:26.603 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920164422 ms)
2025-02-18 11:31:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920164422 ms)
2025-02-18 11:31:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:31:25.401101+00:00', 'runtime': '920160.002', 'adc_vb_f': '12.512', 'adc_va_f': '1.178', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920164423 ms)
2025-02-18 11:31:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:30:25.400174+00:00', 'runtime': '920100.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.178', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920164424 ms)
2025-02-18 11:32:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920224405 ms)
2025-02-18 11:32:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920224405 ms)
2025-02-18 11:32:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:32:25.400575+00:00', 'runtime': '920220.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920224406 ms)
2025-02-18 11:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:31:25.401101+00:00', 'runtime': '920160.002', 'adc_vb_f': '12.512', 'adc_va_f': '1.178', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920224407 ms)
2025-02-18 11:33:07.234 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920265053 ms)
2025-02-18 11:33:07.235 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920265054 ms)
2025-02-18 11:33:07.235 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 33998)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920265054 ms)
2025-02-18 11:33:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920284418 ms)
2025-02-18 11:33:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920284418 ms)
2025-02-18 11:33:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:33:25.402875+00:00', 'runtime': '920280.003', 'adc_vb_f': '12.512', 'adc_va_f': '1.19', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920284419 ms)
2025-02-18 11:33:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:32:25.400575+00:00', 'runtime': '920220.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.181', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920284420 ms)
2025-02-18 11:33:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920293420 ms)
2025-02-18 11:33:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920293421 ms)
2025-02-18 11:33:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:33:25.400152+00:00'} (T+920293422 ms)
2025-02-18 11:34:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920344418 ms)
2025-02-18 11:34:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920344419 ms)
2025-02-18 11:34:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:34:25.402849+00:00', 'runtime': '920340.003', 'adc_vb_f': '12.515', 'adc_va_f': '1.193', 'adc_vl_f': '12.506', 'adc_ic_f': '0.002', 'adc_il_f': '0.449', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920344423 ms)
2025-02-18 11:34:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:33:25.402875+00:00', 'runtime': '920280.003', 'adc_vb_f': '12.512', 'adc_va_f': '1.19', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920344423 ms)
2025-02-18 11:34:48.444 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920366263 ms)
2025-02-18 11:34:48.444 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920366264 ms)
2025-02-18 11:34:48.445 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34106)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920366264 ms)
2025-02-18 11:35:06.540 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920384359 ms)
2025-02-18 11:35:06.540 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920384359 ms)
2025-02-18 11:35:06.540 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34126)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920384359 ms)
2025-02-18 11:35:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920404415 ms)
2025-02-18 11:35:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920404416 ms)
2025-02-18 11:35:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:35:25.400835+00:00', 'runtime': '920400.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.233', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920404420 ms)
2025-02-18 11:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:34:25.402849+00:00', 'runtime': '920340.003', 'adc_vb_f': '12.515', 'adc_va_f': '1.193', 'adc_vl_f': '12.506', 'adc_ic_f': '0.002', 'adc_il_f': '0.449', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.2', 'ahl_t': '1598.2', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '4.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920404420 ms)
2025-02-18 11:36:19.715 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920457534 ms)
2025-02-18 11:36:19.715 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920457534 ms)
2025-02-18 11:36:19.715 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34212)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920457535 ms)
2025-02-18 11:36:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920464414 ms)
2025-02-18 11:36:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920464414 ms)
2025-02-18 11:36:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:36:25.400164+00:00', 'runtime': '920460.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.245', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920464415 ms)
2025-02-18 11:36:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:35:25.400835+00:00', 'runtime': '920400.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.233', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920464416 ms)
2025-02-18 11:36:51.825 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920489645 ms)
2025-02-18 11:36:51.826 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920489645 ms)
2025-02-18 11:36:51.826 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34250)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920489646 ms)
2025-02-18 11:37:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920524413 ms)
2025-02-18 11:37:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920524414 ms)
2025-02-18 11:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:37:25.400159+00:00', 'runtime': '920520.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.263', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920524415 ms)
2025-02-18 11:37:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:36:25.400164+00:00', 'runtime': '920460.001', 'adc_vb_f': '12.518', 'adc_va_f': '1.245', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.506', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920524416 ms)
2025-02-18 11:37:37.944 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920535763 ms)
2025-02-18 11:37:37.944 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920535763 ms)
2025-02-18 11:37:37.944 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34298)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920535764 ms)
2025-02-18 11:38:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920584413 ms)
2025-02-18 11:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920584413 ms)
2025-02-18 11:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:38:25.400259+00:00', 'runtime': '920580.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.285', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920584414 ms)
2025-02-18 11:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:37:25.400159+00:00', 'runtime': '920520.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.263', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.43', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920584415 ms)
2025-02-18 11:39:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920644410 ms)
2025-02-18 11:39:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920644411 ms)
2025-02-18 11:39:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:39:25.400199+00:00', 'runtime': '920640.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.285', 'adc_vl_f': '12.509', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920644412 ms)
2025-02-18 11:39:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:38:25.400259+00:00', 'runtime': '920580.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.285', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920644412 ms)
2025-02-18 11:40:04.194 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920682013 ms)
2025-02-18 11:40:04.194 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920682013 ms)
2025-02-18 11:40:04.194 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34454)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920682014 ms)
2025-02-18 11:40:17.315 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920695134 ms)
2025-02-18 11:40:17.315 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920695134 ms)
2025-02-18 11:40:17.316 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34468)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920695135 ms)
2025-02-18 11:40:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920704411 ms)
2025-02-18 11:40:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920704412 ms)
2025-02-18 11:40:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:40:25.400209+00:00', 'runtime': '920700.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.285', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920704413 ms)
2025-02-18 11:40:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:39:25.400199+00:00', 'runtime': '920640.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.285', 'adc_vl_f': '12.509', 'adc_ic_f': '0.002', 'adc_il_f': '0.457', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920704413 ms)
2025-02-18 11:40:35.765 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920713585 ms)
2025-02-18 11:40:35.766 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920713585 ms)
2025-02-18 11:40:35.767 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:40:25.400338+00:00'} (T+920713586 ms)
2025-02-18 11:41:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920764409 ms)
2025-02-18 11:41:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920764410 ms)
2025-02-18 11:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:41:25.400202+00:00', 'runtime': '920760.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.343', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920764411 ms)
2025-02-18 11:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:40:25.400209+00:00', 'runtime': '920700.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.285', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920764411 ms)
2025-02-18 11:41:35.623 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920773442 ms)
2025-02-18 11:41:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920773442 ms)
2025-02-18 11:41:35.624 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:41:25.400063+00:00'} (T+920773443 ms)
2025-02-18 11:41:55.437 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+920793257 ms)
2025-02-18 11:41:55.438 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+920793257 ms)
2025-02-18 11:41:55.438 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920793258 ms)
2025-02-18 11:41:55.439 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+920793259 ms)
2025-02-18 11:41:55.439 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+920793259 ms)
2025-02-18 11:41:55.440 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920793259 ms)
2025-02-18 11:41:55.440 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+920793260 ms)
2025-02-18 11:41:55.441 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+920793260 ms)
2025-02-18 11:41:55.441 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920793260 ms)
2025-02-18 11:41:55.442 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+920793262 ms)
2025-02-18 11:41:55.443 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+920793262 ms)
2025-02-18 11:41:55.443 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920793262 ms)
2025-02-18 11:41:55.444 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+920793263 ms)
2025-02-18 11:41:55.444 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+920793263 ms)
2025-02-18 11:41:55.444 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920793263 ms)
2025-02-18 11:42:01.516 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920799336 ms)
2025-02-18 11:42:01.517 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920799336 ms)
2025-02-18 11:42:01.517 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34582)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920799337 ms)
2025-02-18 11:42:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920824408 ms)
2025-02-18 11:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920824409 ms)
2025-02-18 11:42:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:42:25.400184+00:00', 'runtime': '920820.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.392', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920824410 ms)
2025-02-18 11:42:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:41:25.400202+00:00', 'runtime': '920760.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.343', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920824410 ms)
2025-02-18 11:42:35.761 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920833580 ms)
2025-02-18 11:42:35.762 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920833581 ms)
2025-02-18 11:42:35.764 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:42:25.400179+00:00'} (T+920833583 ms)
2025-02-18 11:43:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920884413 ms)
2025-02-18 11:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920884413 ms)
2025-02-18 11:43:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:43:25.400214+00:00', 'runtime': '920880.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.392', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920884414 ms)
2025-02-18 11:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:42:25.400184+00:00', 'runtime': '920820.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.392', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.401', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920884415 ms)
2025-02-18 11:43:35.919 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920893738 ms)
2025-02-18 11:43:35.919 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+920893738 ms)
2025-02-18 11:43:35.920 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:43:25.400373+00:00'} (T+920893739 ms)
2025-02-18 11:44:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+920944410 ms)
2025-02-18 11:44:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+920944411 ms)
2025-02-18 11:44:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:44:25.400168+00:00', 'runtime': '920940.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.453', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920944412 ms)
2025-02-18 11:44:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:43:25.400214+00:00', 'runtime': '920880.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.392', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.437', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+920944413 ms)
2025-02-18 11:44:42.791 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+920960610 ms)
2025-02-18 11:44:42.791 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+920960610 ms)
2025-02-18 11:44:42.791 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34754)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920960611 ms)
2025-02-18 11:44:44.794 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+920962614 ms)
2025-02-18 11:44:44.795 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+920962614 ms)
2025-02-18 11:44:44.795 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920962615 ms)
2025-02-18 11:44:44.797 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+920962616 ms)
2025-02-18 11:44:44.797 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+920962616 ms)
2025-02-18 11:44:44.798 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+920962617 ms)
2025-02-18 11:45:25.688 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921003508 ms)
2025-02-18 11:45:25.689 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921003508 ms)
2025-02-18 11:45:25.690 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:45:25.400181+00:00', 'runtime': '921000.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.501', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921003509 ms)
2025-02-18 11:45:25.691 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:44:25.400168+00:00', 'runtime': '920940.001', 'adc_vb_f': '12.515', 'adc_va_f': '1.453', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.399', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921003510 ms)
2025-02-18 11:46:05.990 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921043809 ms)
2025-02-18 11:46:05.990 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921043810 ms)
2025-02-18 11:46:05.991 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34852)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921043810 ms)
2025-02-18 11:46:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921064410 ms)
2025-02-18 11:46:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921064411 ms)
2025-02-18 11:46:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:46:25.400072+00:00', 'runtime': '921060.001', 'adc_vb_f': '12.506', 'adc_va_f': '1.562', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921064412 ms)
2025-02-18 11:46:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:45:25.400181+00:00', 'runtime': '921000.001', 'adc_vb_f': '12.512', 'adc_va_f': '1.501', 'adc_vl_f': '12.509', 'adc_ic_f': '0.005', 'adc_il_f': '0.413', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921064412 ms)
2025-02-18 11:47:24.142 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921121961 ms)
2025-02-18 11:47:24.142 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921121961 ms)
2025-02-18 11:47:24.142 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34934)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921121961 ms)
2025-02-18 11:47:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921124409 ms)
2025-02-18 11:47:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921124409 ms)
2025-02-18 11:47:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:47:25.400166+00:00', 'runtime': '921120.001', 'adc_vb_f': '12.506', 'adc_va_f': '1.608', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921124410 ms)
2025-02-18 11:47:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:46:25.400072+00:00', 'runtime': '921060.001', 'adc_vb_f': '12.506', 'adc_va_f': '1.562', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921124411 ms)
2025-02-18 11:47:36.687 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921134506 ms)
2025-02-18 11:47:36.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921134507 ms)
2025-02-18 11:47:36.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:47:25.400316+00:00'} (T+921134508 ms)
2025-02-18 11:48:23.294 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921181114 ms)
2025-02-18 11:48:23.294 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921181114 ms)
2025-02-18 11:48:23.295 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 34998)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921181114 ms)
2025-02-18 11:48:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921184409 ms)
2025-02-18 11:48:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921184409 ms)
2025-02-18 11:48:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:48:25.400174+00:00', 'runtime': '921180.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.715', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921184411 ms)
2025-02-18 11:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:47:25.400166+00:00', 'runtime': '921120.001', 'adc_vb_f': '12.506', 'adc_va_f': '1.608', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.3', 'ahl_t': '1598.3', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921184411 ms)
2025-02-18 11:48:36.842 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921194661 ms)
2025-02-18 11:48:36.843 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921194662 ms)
2025-02-18 11:48:36.844 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:48:25.400112+00:00'} (T+921194663 ms)
2025-02-18 11:49:04.457 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921222276 ms)
2025-02-18 11:49:04.457 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921222276 ms)
2025-02-18 11:49:04.458 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35044)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921222277 ms)
2025-02-18 11:49:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921244410 ms)
2025-02-18 11:49:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921244411 ms)
2025-02-18 11:49:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:49:25.400171+00:00', 'runtime': '921240.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.822', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921244412 ms)
2025-02-18 11:49:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:48:25.400174+00:00', 'runtime': '921180.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.715', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921244412 ms)
2025-02-18 11:49:35.802 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921253621 ms)
2025-02-18 11:49:35.803 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921253622 ms)
2025-02-18 11:49:35.804 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:49:25.400492+00:00'} (T+921253623 ms)
2025-02-18 11:49:51.594 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921269413 ms)
2025-02-18 11:49:51.594 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921269414 ms)
2025-02-18 11:49:51.595 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35096)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921269414 ms)
2025-02-18 11:50:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921304410 ms)
2025-02-18 11:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921304411 ms)
2025-02-18 11:50:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:50:25.400020+00:00', 'runtime': '921300.001', 'adc_vb_f': '12.506', 'adc_va_f': '1.929', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921304412 ms)
2025-02-18 11:50:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:49:25.400171+00:00', 'runtime': '921240.001', 'adc_vb_f': '12.509', 'adc_va_f': '1.822', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921304412 ms)
2025-02-18 11:50:35.607 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921313426 ms)
2025-02-18 11:50:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921313427 ms)
2025-02-18 11:50:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:50:25.409124+00:00'} (T+921313428 ms)
2025-02-18 11:51:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921364410 ms)
2025-02-18 11:51:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921364411 ms)
2025-02-18 11:51:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:51:25.402812+00:00', 'runtime': '921360.003', 'adc_vb_f': '12.506', 'adc_va_f': '2.036', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921364415 ms)
2025-02-18 11:51:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:50:25.400020+00:00', 'runtime': '921300.001', 'adc_vb_f': '12.506', 'adc_va_f': '1.929', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921364415 ms)
2025-02-18 11:51:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921373433 ms)
2025-02-18 11:51:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921373433 ms)
2025-02-18 11:51:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:51:25.400206+00:00'} (T+921373435 ms)
2025-02-18 11:52:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921424408 ms)
2025-02-18 11:52:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921424409 ms)
2025-02-18 11:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:52:25.400244+00:00', 'runtime': '921420.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.249', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921424413 ms)
2025-02-18 11:52:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:51:25.402812+00:00', 'runtime': '921360.003', 'adc_vb_f': '12.506', 'adc_va_f': '2.036', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.423', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.503', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921424414 ms)
2025-02-18 11:52:35.812 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921433631 ms)
2025-02-18 11:52:35.812 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921433632 ms)
2025-02-18 11:52:35.813 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:52:25.413126+00:00'} (T+921433633 ms)
2025-02-18 11:52:56.919 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921454738 ms)
2025-02-18 11:52:56.919 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921454739 ms)
2025-02-18 11:52:56.920 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35294)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921454739 ms)
2025-02-18 11:53:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921484411 ms)
2025-02-18 11:53:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921484411 ms)
2025-02-18 11:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:53:25.400153+00:00', 'runtime': '921480.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.423', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921484412 ms)
2025-02-18 11:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:52:25.400244+00:00', 'runtime': '921420.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.249', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.445', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921484413 ms)
2025-02-18 11:53:35.716 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921493535 ms)
2025-02-18 11:53:35.717 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921493536 ms)
2025-02-18 11:53:35.718 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:53:25.399998+00:00'} (T+921493537 ms)
2025-02-18 11:53:55.068 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921512887 ms)
2025-02-18 11:53:55.068 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921512887 ms)
2025-02-18 11:53:55.069 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35358)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921512888 ms)
2025-02-18 11:54:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921544410 ms)
2025-02-18 11:54:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921544411 ms)
2025-02-18 11:54:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:54:25.400164+00:00', 'runtime': '921540.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.625', 'adc_vl_f': '12.485', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921544412 ms)
2025-02-18 11:54:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:53:25.400153+00:00', 'runtime': '921480.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.423', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.509', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20142', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921544412 ms)
2025-02-18 11:54:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921553423 ms)
2025-02-18 11:54:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921553424 ms)
2025-02-18 11:54:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:54:25.400072+00:00'} (T+921553425 ms)
2025-02-18 11:55:24.260 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921602079 ms)
2025-02-18 11:55:24.260 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921602079 ms)
2025-02-18 11:55:24.261 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35452)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921602080 ms)
2025-02-18 11:55:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921604411 ms)
2025-02-18 11:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921604411 ms)
2025-02-18 11:55:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:55:25.400209+00:00', 'runtime': '921600.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.853', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921604412 ms)
2025-02-18 11:55:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:54:25.400164+00:00', 'runtime': '921540.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.625', 'adc_vl_f': '12.485', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921604412 ms)
2025-02-18 11:55:36.203 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921614022 ms)
2025-02-18 11:55:36.204 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921614023 ms)
2025-02-18 11:55:36.205 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:55:25.400224+00:00'} (T+921614024 ms)
2025-02-18 11:55:41.350 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921619169 ms)
2025-02-18 11:55:41.350 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921619169 ms)
2025-02-18 11:55:41.351 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35472)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921619170 ms)
2025-02-18 11:55:48.420 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921626239 ms)
2025-02-18 11:55:48.420 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921626240 ms)
2025-02-18 11:55:48.421 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35490)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921626240 ms)
2025-02-18 11:56:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921664412 ms)
2025-02-18 11:56:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921664412 ms)
2025-02-18 11:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:56:25.400180+00:00', 'runtime': '921660.001', 'adc_vb_f': '12.506', 'adc_va_f': '3.11', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921664413 ms)
2025-02-18 11:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:55:25.400209+00:00', 'runtime': '921600.001', 'adc_vb_f': '12.506', 'adc_va_f': '2.853', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.459', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.5', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921664414 ms)
2025-02-18 11:57:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921724408 ms)
2025-02-18 11:57:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921724409 ms)
2025-02-18 11:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:57:25.400177+00:00', 'runtime': '921720.001', 'adc_vb_f': '12.506', 'adc_va_f': '3.421', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921724410 ms)
2025-02-18 11:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:56:25.400180+00:00', 'runtime': '921660.001', 'adc_vb_f': '12.506', 'adc_va_f': '3.11', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921724410 ms)
2025-02-18 11:58:00.645 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921758464 ms)
2025-02-18 11:58:00.645 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921758465 ms)
2025-02-18 11:58:00.646 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35632)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921758465 ms)
2025-02-18 11:58:17.743 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921775562 ms)
2025-02-18 11:58:17.743 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921775562 ms)
2025-02-18 11:58:17.744 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35650)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921775563 ms)
2025-02-18 11:58:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921784408 ms)
2025-02-18 11:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921784408 ms)
2025-02-18 11:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:58:25.400179+00:00', 'runtime': '921780.001', 'adc_vb_f': '12.506', 'adc_va_f': '3.754', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921784411 ms)
2025-02-18 11:58:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:57:25.400177+00:00', 'runtime': '921720.001', 'adc_vb_f': '12.506', 'adc_va_f': '3.421', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.406', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921784411 ms)
2025-02-18 11:58:35.812 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921793631 ms)
2025-02-18 11:58:35.812 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921793632 ms)
2025-02-18 11:58:35.814 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:58:25.400337+00:00'} (T+921793633 ms)
2025-02-18 11:59:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921844406 ms)
2025-02-18 11:59:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921844406 ms)
2025-02-18 11:59:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 11:59:25.400182+00:00', 'runtime': '921840.001', 'adc_vb_f': '12.503', 'adc_va_f': '4.181', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921844407 ms)
2025-02-18 11:59:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:58:25.400179+00:00', 'runtime': '921780.001', 'adc_vb_f': '12.506', 'adc_va_f': '3.754', 'adc_vl_f': '12.497', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921844408 ms)
2025-02-18 11:59:35.725 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921853545 ms)
2025-02-18 11:59:35.726 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+921853545 ms)
2025-02-18 11:59:35.727 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 11:59:25.399992+00:00'} (T+921853546 ms)
2025-02-18 12:00:25.635 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921903454 ms)
2025-02-18 12:00:25.635 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921903455 ms)
2025-02-18 12:00:25.636 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:00:25.400203+00:00', 'runtime': '921900.001', 'adc_vb_f': '12.503', 'adc_va_f': '4.669', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.494', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921903455 ms)
2025-02-18 12:00:25.637 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 11:59:25.400182+00:00', 'runtime': '921840.001', 'adc_vb_f': '12.503', 'adc_va_f': '4.181', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.497', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921903456 ms)
2025-02-18 12:00:29.974 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+921907793 ms)
2025-02-18 12:00:29.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+921907793 ms)
2025-02-18 12:00:29.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35792)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921907794 ms)
2025-02-18 12:01:06.021 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+921943841 ms)
2025-02-18 12:01:06.022 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+921943841 ms)
2025-02-18 12:01:06.022 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921943842 ms)
2025-02-18 12:01:06.024 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+921943843 ms)
2025-02-18 12:01:06.024 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+921943843 ms)
2025-02-18 12:01:06.024 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+921943844 ms)
2025-02-18 12:01:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+921964407 ms)
2025-02-18 12:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+921964407 ms)
2025-02-18 12:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:01:25.400173+00:00', 'runtime': '921960.001', 'adc_vb_f': '12.506', 'adc_va_f': '5.243', 'adc_vl_f': '12.479', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.494', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921964408 ms)
2025-02-18 12:01:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:00:25.400203+00:00', 'runtime': '921900.001', 'adc_vb_f': '12.503', 'adc_va_f': '4.669', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.486', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.494', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+921964409 ms)
2025-02-18 12:02:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922024407 ms)
2025-02-18 12:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922024408 ms)
2025-02-18 12:02:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:02:25.400216+00:00', 'runtime': '922020.001', 'adc_vb_f': '12.506', 'adc_va_f': '5.875', 'adc_vl_f': '12.476', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.494', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922024409 ms)
2025-02-18 12:02:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:01:25.400173+00:00', 'runtime': '921960.001', 'adc_vb_f': '12.506', 'adc_va_f': '5.243', 'adc_vl_f': '12.479', 'adc_ic_f': '0.005', 'adc_il_f': '0.466', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.4', 'ahl_t': '1598.4', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.494', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922024409 ms)
2025-02-18 12:02:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922033430 ms)
2025-02-18 12:02:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922033430 ms)
2025-02-18 12:02:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:02:25.400339+00:00'} (T+922033431 ms)
2025-02-18 12:03:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922084407 ms)
2025-02-18 12:03:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922084408 ms)
2025-02-18 12:03:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:03:25.400184+00:00', 'runtime': '922080.001', 'adc_vb_f': '12.503', 'adc_va_f': '6.54', 'adc_vl_f': '12.479', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.485', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922084409 ms)
2025-02-18 12:03:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:02:25.400216+00:00', 'runtime': '922020.001', 'adc_vb_f': '12.506', 'adc_va_f': '5.875', 'adc_vl_f': '12.476', 'adc_ic_f': '0.005', 'adc_il_f': '0.476', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.494', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922084409 ms)
2025-02-18 12:03:35.292 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922093112 ms)
2025-02-18 12:03:35.293 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922093112 ms)
2025-02-18 12:03:35.293 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 35990)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922093112 ms)
2025-02-18 12:03:36.051 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922093870 ms)
2025-02-18 12:03:36.052 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922093871 ms)
2025-02-18 12:03:36.053 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:03:25.400507+00:00'} (T+922093872 ms)
2025-02-18 12:04:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922144407 ms)
2025-02-18 12:04:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922144407 ms)
2025-02-18 12:04:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:04:25.400192+00:00', 'runtime': '922140.001', 'adc_vb_f': '12.506', 'adc_va_f': '7.291', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.485', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922144408 ms)
2025-02-18 12:04:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:03:25.400184+00:00', 'runtime': '922080.001', 'adc_vb_f': '12.503', 'adc_va_f': '6.54', 'adc_vl_f': '12.479', 'adc_ic_f': '0.005', 'adc_il_f': '0.464', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.485', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922144409 ms)
2025-02-18 12:04:30.473 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922148292 ms)
2025-02-18 12:04:30.473 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922148292 ms)
2025-02-18 12:04:30.474 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36048)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922148293 ms)
2025-02-18 12:04:35.583 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922153402 ms)
2025-02-18 12:04:35.584 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922153403 ms)
2025-02-18 12:04:35.585 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:04:25.400266+00:00'} (T+922153404 ms)
2025-02-18 12:05:06.565 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922184384 ms)
2025-02-18 12:05:06.565 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922184385 ms)
2025-02-18 12:05:06.566 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36090)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922184385 ms)
2025-02-18 12:05:11.645 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922189465 ms)
2025-02-18 12:05:11.646 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922189465 ms)
2025-02-18 12:05:11.646 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36096)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922189466 ms)
2025-02-18 12:05:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922204408 ms)
2025-02-18 12:05:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922204408 ms)
2025-02-18 12:05:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:05:25.400192+00:00', 'runtime': '922200.001', 'adc_vb_f': '12.506', 'adc_va_f': '8.041', 'adc_vl_f': '12.494', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.485', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922204409 ms)
2025-02-18 12:05:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:04:25.400192+00:00', 'runtime': '922140.001', 'adc_vb_f': '12.506', 'adc_va_f': '7.291', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.447', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.485', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922204410 ms)
2025-02-18 12:05:35.757 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922213577 ms)
2025-02-18 12:05:35.758 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922213577 ms)
2025-02-18 12:05:35.759 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:05:25.400234+00:00'} (T+922213578 ms)
2025-02-18 12:06:25.637 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922263457 ms)
2025-02-18 12:06:25.638 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922263457 ms)
2025-02-18 12:06:25.642 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:06:25.400042+00:00', 'runtime': '922260.001', 'adc_vb_f': '12.506', 'adc_va_f': '8.795', 'adc_vl_f': '12.485', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.503', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922263461 ms)
2025-02-18 12:06:25.642 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:05:25.400192+00:00', 'runtime': '922200.001', 'adc_vb_f': '12.506', 'adc_va_f': '8.041', 'adc_vl_f': '12.494', 'adc_ic_f': '0.005', 'adc_il_f': '0.416', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.485', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922263461 ms)
2025-02-18 12:06:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922273437 ms)
2025-02-18 12:06:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922273438 ms)
2025-02-18 12:06:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:06:25.409121+00:00'} (T+922273438 ms)
2025-02-18 12:06:44.806 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922282625 ms)
2025-02-18 12:06:44.806 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922282626 ms)
2025-02-18 12:06:44.807 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36206)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922282626 ms)
2025-02-18 12:06:46.885 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922284705 ms)
2025-02-18 12:06:46.886 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922284705 ms)
2025-02-18 12:06:46.886 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36210)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922284706 ms)
2025-02-18 12:07:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922324406 ms)
2025-02-18 12:07:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922324407 ms)
2025-02-18 12:07:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:07:25.400135+00:00', 'runtime': '922320.001', 'adc_vb_f': '12.491', 'adc_va_f': '9.555', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922324408 ms)
2025-02-18 12:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:06:25.400042+00:00', 'runtime': '922260.001', 'adc_vb_f': '12.506', 'adc_va_f': '8.795', 'adc_vl_f': '12.485', 'adc_ic_f': '0.005', 'adc_il_f': '0.42', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.503', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922324411 ms)
2025-02-18 12:07:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922333434 ms)
2025-02-18 12:07:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922333434 ms)
2025-02-18 12:07:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:07:25.413128+00:00'} (T+922333435 ms)
2025-02-18 12:08:25.635 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922383454 ms)
2025-02-18 12:08:25.635 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922383455 ms)
2025-02-18 12:08:25.636 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:08:25.402804+00:00', 'runtime': '922380.003', 'adc_vb_f': '12.491', 'adc_va_f': '10.364', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922383456 ms)
2025-02-18 12:08:25.637 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:07:25.400135+00:00', 'runtime': '922320.001', 'adc_vb_f': '12.491', 'adc_va_f': '9.555', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922383456 ms)
2025-02-18 12:08:35.620 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922393439 ms)
2025-02-18 12:08:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922393440 ms)
2025-02-18 12:08:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:08:25.400174+00:00'} (T+922393444 ms)
2025-02-18 12:09:26.607 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922444426 ms)
2025-02-18 12:09:26.607 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922444426 ms)
2025-02-18 12:09:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:09:25.400204+00:00', 'runtime': '922440.001', 'adc_vb_f': '12.491', 'adc_va_f': '11.154', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922444427 ms)
2025-02-18 12:09:26.608 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:08:25.402804+00:00', 'runtime': '922380.003', 'adc_vb_f': '12.491', 'adc_va_f': '10.364', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.483', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922444428 ms)
2025-02-18 12:09:35.621 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922453441 ms)
2025-02-18 12:09:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922453441 ms)
2025-02-18 12:09:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:09:25.413128+00:00'} (T+922453442 ms)
2025-02-18 12:10:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922504421 ms)
2025-02-18 12:10:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922504422 ms)
2025-02-18 12:10:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:10:25.400138+00:00', 'runtime': '922500.001', 'adc_vb_f': '12.491', 'adc_va_f': '11.798', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922504423 ms)
2025-02-18 12:10:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:09:25.400204+00:00', 'runtime': '922440.001', 'adc_vb_f': '12.491', 'adc_va_f': '11.154', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.473', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922504423 ms)
2025-02-18 12:10:35.630 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922513449 ms)
2025-02-18 12:10:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922513450 ms)
2025-02-18 12:10:35.632 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:10:25.413122+00:00'} (T+922513451 ms)
2025-02-18 12:11:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922564422 ms)
2025-02-18 12:11:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922564422 ms)
2025-02-18 12:11:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:11:25.402857+00:00', 'runtime': '922560.003', 'adc_vb_f': '12.494', 'adc_va_f': '11.905', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922564424 ms)
2025-02-18 12:11:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:10:25.400138+00:00', 'runtime': '922500.001', 'adc_vb_f': '12.491', 'adc_va_f': '11.798', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.461', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922564424 ms)
2025-02-18 12:11:35.680 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922573499 ms)
2025-02-18 12:11:35.684 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922573503 ms)
2025-02-18 12:11:35.685 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:11:25.400260+00:00'} (T+922573504 ms)
2025-02-18 12:12:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922624421 ms)
2025-02-18 12:12:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922624422 ms)
2025-02-18 12:12:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:12:25.402723+00:00', 'runtime': '922620.003', 'adc_vb_f': '12.494', 'adc_va_f': '12.506', 'adc_vl_f': '12.482', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922624424 ms)
2025-02-18 12:12:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:11:25.402857+00:00', 'runtime': '922560.003', 'adc_vb_f': '12.494', 'adc_va_f': '11.905', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.469', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922624424 ms)
2025-02-18 12:12:36.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922634419 ms)
2025-02-18 12:12:36.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922634420 ms)
2025-02-18 12:12:36.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:12:25.400136+00:00'} (T+922634421 ms)
2025-02-18 12:13:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922684420 ms)
2025-02-18 12:13:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922684421 ms)
2025-02-18 12:13:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:13:25.400166+00:00', 'runtime': '922680.001', 'adc_vb_f': '12.5', 'adc_va_f': '13.083', 'adc_vl_f': '12.491', 'adc_ic_f': '0.0', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.138', 'sweep_pmax': '5.013', 'sweep_voc': '13.513', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922684423 ms)
2025-02-18 12:13:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:12:25.402723+00:00', 'runtime': '922620.003', 'adc_vb_f': '12.494', 'adc_va_f': '12.506', 'adc_vl_f': '12.482', 'adc_ic_f': '0.005', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '3', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '0.0', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '8.447', 'sweep_pmax': '5.013', 'sweep_voc': '11.264', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922684423 ms)
2025-02-18 12:13:35.699 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922693518 ms)
2025-02-18 12:13:35.699 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922693519 ms)
2025-02-18 12:13:35.700 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:13:25.411708+00:00'} (T+922693520 ms)
2025-02-18 12:14:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922744421 ms)
2025-02-18 12:14:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922744421 ms)
2025-02-18 12:14:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:14:25.401132+00:00', 'runtime': '922740.002', 'adc_vb_f': '12.503', 'adc_va_f': '13.138', 'adc_vl_f': '12.497', 'adc_ic_f': '0.002', 'adc_il_f': '0.403', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '10.553', 'sweep_pmax': '5.013', 'sweep_voc': '14.069', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922744425 ms)
2025-02-18 12:14:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:13:25.400166+00:00', 'runtime': '922680.001', 'adc_vb_f': '12.5', 'adc_va_f': '13.083', 'adc_vl_f': '12.491', 'adc_ic_f': '0.0', 'adc_il_f': '0.44', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.0', 'sweep_vmp': '10.138', 'sweep_pmax': '5.013', 'sweep_voc': '13.513', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922744426 ms)
2025-02-18 12:14:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922753428 ms)
2025-02-18 12:14:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922753428 ms)
2025-02-18 12:14:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:14:25.400092+00:00'} (T+922753429 ms)
2025-02-18 12:14:55.544 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922773364 ms)
2025-02-18 12:14:55.544 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922773364 ms)
2025-02-18 12:14:55.545 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36734)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922773364 ms)
2025-02-18 12:15:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922804421 ms)
2025-02-18 12:15:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922804421 ms)
2025-02-18 12:15:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:15:25.400147+00:00', 'runtime': '922800.001', 'adc_vb_f': '12.5', 'adc_va_f': '13.174', 'adc_vl_f': '12.494', 'adc_ic_f': '0.007', 'adc_il_f': '0.418', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '11.365', 'sweep_pmax': '5.013', 'sweep_voc': '15.152', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922804422 ms)
2025-02-18 12:15:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:14:25.401132+00:00', 'runtime': '922740.002', 'adc_vb_f': '12.503', 'adc_va_f': '13.138', 'adc_vl_f': '12.497', 'adc_ic_f': '0.002', 'adc_il_f': '0.403', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '10.553', 'sweep_pmax': '5.013', 'sweep_voc': '14.069', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922804423 ms)
2025-02-18 12:15:46.657 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922824477 ms)
2025-02-18 12:15:46.658 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922824477 ms)
2025-02-18 12:15:46.658 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36796)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922824477 ms)
2025-02-18 12:16:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922864415 ms)
2025-02-18 12:16:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922864416 ms)
2025-02-18 12:16:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:16:25.400172+00:00', 'runtime': '922860.001', 'adc_vb_f': '12.506', 'adc_va_f': '13.193', 'adc_vl_f': '12.503', 'adc_ic_f': '0.012', 'adc_il_f': '0.391', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '11.987', 'sweep_pmax': '5.013', 'sweep_voc': '15.979', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922864417 ms)
2025-02-18 12:16:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:15:25.400147+00:00', 'runtime': '922800.001', 'adc_vb_f': '12.5', 'adc_va_f': '13.174', 'adc_vl_f': '12.494', 'adc_ic_f': '0.007', 'adc_il_f': '0.418', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.5', 'ahl_t': '1598.5', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '11.365', 'sweep_pmax': '5.013', 'sweep_voc': '15.152', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922864417 ms)
2025-02-18 12:16:55.836 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+922893655 ms)
2025-02-18 12:16:55.836 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+922893656 ms)
2025-02-18 12:16:55.837 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 36872)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+922893656 ms)
2025-02-18 12:17:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922924417 ms)
2025-02-18 12:17:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922924418 ms)
2025-02-18 12:17:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:17:25.400172+00:00', 'runtime': '922920.001', 'adc_vb_f': '12.494', 'adc_va_f': '13.193', 'adc_vl_f': '12.476', 'adc_ic_f': '0.012', 'adc_il_f': '0.464', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.136', 'sweep_vmp': '12.463', 'sweep_pmax': '5.013', 'sweep_voc': '16.614', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922924419 ms)
2025-02-18 12:17:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:16:25.400172+00:00', 'runtime': '922860.001', 'adc_vb_f': '12.506', 'adc_va_f': '13.193', 'adc_vl_f': '12.503', 'adc_ic_f': '0.012', 'adc_il_f': '0.391', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '11.987', 'sweep_pmax': '5.013', 'sweep_voc': '15.979', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922924419 ms)
2025-02-18 12:18:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+922984417 ms)
2025-02-18 12:18:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+922984418 ms)
2025-02-18 12:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:18:25.400168+00:00', 'runtime': '922980.001', 'adc_vb_f': '12.497', 'adc_va_f': '13.193', 'adc_vl_f': '12.482', 'adc_ic_f': '0.017', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.5', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.196', 'sweep_vmp': '13.04', 'sweep_pmax': '5.013', 'sweep_voc': '17.386', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922984419 ms)
2025-02-18 12:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:17:25.400172+00:00', 'runtime': '922920.001', 'adc_vb_f': '12.494', 'adc_va_f': '13.193', 'adc_vl_f': '12.476', 'adc_ic_f': '0.012', 'adc_il_f': '0.464', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.136', 'sweep_vmp': '12.463', 'sweep_pmax': '5.013', 'sweep_voc': '16.614', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+922984419 ms)
2025-02-18 12:18:35.597 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922993416 ms)
2025-02-18 12:18:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+922993417 ms)
2025-02-18 12:18:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:18:25.413052+00:00'} (T+922993418 ms)
2025-02-18 12:19:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923044419 ms)
2025-02-18 12:19:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923044419 ms)
2025-02-18 12:19:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:19:25.400240+00:00', 'runtime': '923040.001', 'adc_vb_f': '12.497', 'adc_va_f': '13.571', 'adc_vl_f': '12.476', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.5', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '13.528', 'sweep_pmax': '5.013', 'sweep_voc': '18.036', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923044423 ms)
2025-02-18 12:19:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:18:25.400168+00:00', 'runtime': '922980.001', 'adc_vb_f': '12.497', 'adc_va_f': '13.193', 'adc_vl_f': '12.482', 'adc_ic_f': '0.017', 'adc_il_f': '0.452', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.5', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.196', 'sweep_vmp': '13.04', 'sweep_pmax': '5.013', 'sweep_voc': '17.386', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923044424 ms)
2025-02-18 12:20:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923104417 ms)
2025-02-18 12:20:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923104417 ms)
2025-02-18 12:20:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:20:25.401171+00:00', 'runtime': '923100.002', 'adc_vb_f': '12.515', 'adc_va_f': '13.962', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '13.919', 'sweep_pmax': '5.013', 'sweep_voc': '18.555', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923104418 ms)
2025-02-18 12:20:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:19:25.400240+00:00', 'runtime': '923040.001', 'adc_vb_f': '12.497', 'adc_va_f': '13.571', 'adc_vl_f': '12.476', 'adc_ic_f': '0.005', 'adc_il_f': '0.471', 't_hs': '2', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.5', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '13.528', 'sweep_pmax': '5.013', 'sweep_voc': '18.036', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923104418 ms)
2025-02-18 12:21:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923164416 ms)
2025-02-18 12:21:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923164417 ms)
2025-02-18 12:21:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:21:25.400157+00:00', 'runtime': '923160.001', 'adc_vb_f': '12.515', 'adc_va_f': '14.108', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.37', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '14.075', 'sweep_pmax': '5.013', 'sweep_voc': '18.762', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923164421 ms)
2025-02-18 12:21:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:20:25.401171+00:00', 'runtime': '923100.002', 'adc_vb_f': '12.515', 'adc_va_f': '13.962', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.442', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.493', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '13.919', 'sweep_pmax': '5.013', 'sweep_voc': '18.555', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923164421 ms)
2025-02-18 12:22:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923224415 ms)
2025-02-18 12:22:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923224416 ms)
2025-02-18 12:22:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:22:25.400173+00:00', 'runtime': '923220.001', 'adc_vb_f': '12.503', 'adc_va_f': '14.255', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '14.23', 'sweep_pmax': '5.013', 'sweep_voc': '18.97', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923224420 ms)
2025-02-18 12:22:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:21:25.400157+00:00', 'runtime': '923160.001', 'adc_vb_f': '12.515', 'adc_va_f': '14.108', 'adc_vl_f': '12.469', 'adc_ic_f': '0.005', 'adc_il_f': '0.37', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '14.075', 'sweep_pmax': '5.013', 'sweep_voc': '18.762', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923224421 ms)
2025-02-18 12:22:35.628 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923233447 ms)
2025-02-18 12:22:35.628 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923233448 ms)
2025-02-18 12:22:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:22:25.400343+00:00'} (T+923233449 ms)
2025-02-18 12:23:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923284413 ms)
2025-02-18 12:23:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923284414 ms)
2025-02-18 12:23:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:23:25.400179+00:00', 'runtime': '923280.001', 'adc_vb_f': '12.494', 'adc_va_f': '14.398', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.355', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.359', 'sweep_pmax': '5.013', 'sweep_voc': '19.144', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923284418 ms)
2025-02-18 12:23:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:22:25.400173+00:00', 'runtime': '923220.001', 'adc_vb_f': '12.503', 'adc_va_f': '14.255', 'adc_vl_f': '12.473', 'adc_ic_f': '0.005', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '14.23', 'sweep_pmax': '5.013', 'sweep_voc': '18.97', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923284418 ms)
2025-02-18 12:23:35.704 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923293523 ms)
2025-02-18 12:23:35.705 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923293524 ms)
2025-02-18 12:23:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:23:25.400182+00:00'} (T+923293525 ms)
2025-02-18 12:24:04.365 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923322185 ms)
2025-02-18 12:24:04.366 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923322185 ms)
2025-02-18 12:24:04.366 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923322186 ms)
2025-02-18 12:24:04.367 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923322186 ms)
2025-02-18 12:24:04.367 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923322187 ms)
2025-02-18 12:24:04.368 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923322187 ms)
2025-02-18 12:24:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923344412 ms)
2025-02-18 12:24:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923344413 ms)
2025-02-18 12:24:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:24:25.400200+00:00', 'runtime': '923340.001', 'adc_vb_f': '12.506', 'adc_va_f': '14.398', 'adc_vl_f': '12.491', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '14.407', 'sweep_pmax': '5.013', 'sweep_voc': '19.208', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923344414 ms)
2025-02-18 12:24:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:23:25.400179+00:00', 'runtime': '923280.001', 'adc_vb_f': '12.494', 'adc_va_f': '14.398', 'adc_vl_f': '12.5', 'adc_ic_f': '0.005', 'adc_il_f': '0.355', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.359', 'sweep_pmax': '5.013', 'sweep_voc': '19.144', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923344414 ms)
2025-02-18 12:25:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923404410 ms)
2025-02-18 12:25:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923404411 ms)
2025-02-18 12:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:25:25.400166+00:00', 'runtime': '923400.001', 'adc_vb_f': '12.5', 'adc_va_f': '14.45', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.316', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.491', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '14.407', 'sweep_pmax': '5.013', 'sweep_voc': '19.208', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923404412 ms)
2025-02-18 12:25:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:24:25.400200+00:00', 'runtime': '923340.001', 'adc_vb_f': '12.506', 'adc_va_f': '14.398', 'adc_vl_f': '12.491', 'adc_ic_f': '0.005', 'adc_il_f': '0.435', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.03', 'sweep_vmp': '14.407', 'sweep_pmax': '5.013', 'sweep_voc': '19.208', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923404413 ms)
2025-02-18 12:26:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923464410 ms)
2025-02-18 12:26:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923464410 ms)
2025-02-18 12:26:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:26:25.400168+00:00', 'runtime': '923460.001', 'adc_vb_f': '12.497', 'adc_va_f': '14.447', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.319', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.491', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.438', 'sweep_pmax': '5.013', 'sweep_voc': '19.247', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923464411 ms)
2025-02-18 12:26:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:25:25.400166+00:00', 'runtime': '923400.001', 'adc_vb_f': '12.5', 'adc_va_f': '14.45', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.316', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.491', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.06', 'sweep_vmp': '14.407', 'sweep_pmax': '5.013', 'sweep_voc': '19.208', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923464412 ms)
2025-02-18 12:26:35.819 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923473638 ms)
2025-02-18 12:26:35.820 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923473639 ms)
2025-02-18 12:26:35.820 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:26:25.400328+00:00'} (T+923473640 ms)
2025-02-18 12:27:22.609 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923520429 ms)
2025-02-18 12:27:22.610 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923520429 ms)
2025-02-18 12:27:22.610 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923520430 ms)
2025-02-18 12:27:22.611 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923520431 ms)
2025-02-18 12:27:22.611 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923520431 ms)
2025-02-18 12:27:22.612 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923520431 ms)
2025-02-18 12:27:22.612 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923520432 ms)
2025-02-18 12:27:22.613 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923520432 ms)
2025-02-18 12:27:22.613 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923520432 ms)
2025-02-18 12:27:22.614 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923520433 ms)
2025-02-18 12:27:22.614 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923520433 ms)
2025-02-18 12:27:22.614 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923520433 ms)
2025-02-18 12:27:22.616 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923520435 ms)
2025-02-18 12:27:22.616 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923520435 ms)
2025-02-18 12:27:22.616 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923520435 ms)
2025-02-18 12:27:22.617 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+923520436 ms)
2025-02-18 12:27:22.617 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+923520436 ms)
2025-02-18 12:27:22.619 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923520438 ms)
2025-02-18 12:27:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923524409 ms)
2025-02-18 12:27:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923524410 ms)
2025-02-18 12:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:27:25.400179+00:00', 'runtime': '923520.001', 'adc_vb_f': '12.488', 'adc_va_f': '14.398', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.372', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.488', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.401', 'sweep_pmax': '5.013', 'sweep_voc': '19.199', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923524411 ms)
2025-02-18 12:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:26:25.400168+00:00', 'runtime': '923460.001', 'adc_vb_f': '12.497', 'adc_va_f': '14.447', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.319', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.491', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.438', 'sweep_pmax': '5.013', 'sweep_voc': '19.247', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923524411 ms)
2025-02-18 12:28:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923584407 ms)
2025-02-18 12:28:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923584407 ms)
2025-02-18 12:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:28:25.400193+00:00', 'runtime': '923580.001', 'adc_vb_f': '12.5', 'adc_va_f': '14.493', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.309', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '5.013', 'sweep_voc': '19.299', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923584409 ms)
2025-02-18 12:28:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:27:25.400179+00:00', 'runtime': '923520.001', 'adc_vb_f': '12.488', 'adc_va_f': '14.398', 'adc_vl_f': '12.488', 'adc_ic_f': '0.005', 'adc_il_f': '0.372', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.488', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.401', 'sweep_pmax': '5.013', 'sweep_voc': '19.199', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923584409 ms)
2025-02-18 12:29:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923644408 ms)
2025-02-18 12:29:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923644408 ms)
2025-02-18 12:29:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:29:25.400266+00:00', 'runtime': '923640.001', 'adc_vb_f': '12.5', 'adc_va_f': '14.603', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.367', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '14.56', 'sweep_pmax': '5.013', 'sweep_voc': '19.409', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923644409 ms)
2025-02-18 12:29:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:28:25.400193+00:00', 'runtime': '923580.001', 'adc_vb_f': '12.5', 'adc_va_f': '14.493', 'adc_vl_f': '12.506', 'adc_ic_f': '0.005', 'adc_il_f': '0.309', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.494', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.015', 'sweep_vmp': '14.478', 'sweep_pmax': '5.013', 'sweep_voc': '19.299', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923644410 ms)
2025-02-18 12:29:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923653418 ms)
2025-02-18 12:29:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+923653418 ms)
2025-02-18 12:29:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:29:25.400345+00:00'} (T+923653420 ms)
2025-02-18 12:30:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923704406 ms)
2025-02-18 12:30:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923704407 ms)
2025-02-18 12:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:30:25.400176+00:00', 'runtime': '923700.001', 'adc_vb_f': '12.497', 'adc_va_f': '14.749', 'adc_vl_f': '12.506', 'adc_ic_f': '0.007', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.5', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '14.716', 'sweep_pmax': '5.013', 'sweep_voc': '19.617', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923704408 ms)
2025-02-18 12:30:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:29:25.400266+00:00', 'runtime': '923640.001', 'adc_vb_f': '12.5', 'adc_va_f': '14.603', 'adc_vl_f': '12.503', 'adc_ic_f': '0.005', 'adc_il_f': '0.367', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.497', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.045', 'sweep_vmp': '14.56', 'sweep_pmax': '5.013', 'sweep_voc': '19.409', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923704408 ms)
2025-02-18 12:31:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923764406 ms)
2025-02-18 12:31:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923764407 ms)
2025-02-18 12:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:31:25.400170+00:00', 'runtime': '923760.001', 'adc_vb_f': '12.503', 'adc_va_f': '14.844', 'adc_vl_f': '12.509', 'adc_ic_f': '0.01', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.503', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '14.81', 'sweep_pmax': '5.013', 'sweep_voc': '19.745', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923764408 ms)
2025-02-18 12:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:30:25.400176+00:00', 'runtime': '923700.001', 'adc_vb_f': '12.497', 'adc_va_f': '14.749', 'adc_vl_f': '12.506', 'adc_ic_f': '0.007', 'adc_il_f': '0.43', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.5', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.6', 'ahl_t': '1598.6', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.091', 'sweep_vmp': '14.716', 'sweep_pmax': '5.013', 'sweep_voc': '19.617', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923764408 ms)
2025-02-18 12:32:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923824406 ms)
2025-02-18 12:32:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923824406 ms)
2025-02-18 12:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:32:25.400605+00:00', 'runtime': '923820.001', 'adc_vb_f': '12.503', 'adc_va_f': '14.987', 'adc_vl_f': '12.509', 'adc_ic_f': '0.019', 'adc_il_f': '0.401', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.211', 'sweep_vmp': '14.963', 'sweep_pmax': '5.013', 'sweep_voc': '19.946', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923824407 ms)
2025-02-18 12:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:31:25.400170+00:00', 'runtime': '923760.001', 'adc_vb_f': '12.503', 'adc_va_f': '14.844', 'adc_vl_f': '12.509', 'adc_ic_f': '0.01', 'adc_il_f': '0.406', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.503', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.121', 'sweep_vmp': '14.81', 'sweep_pmax': '5.013', 'sweep_voc': '19.745', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923824408 ms)
2025-02-18 12:32:39.051 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+923836870 ms)
2025-02-18 12:32:39.051 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+923836870 ms)
2025-02-18 12:32:39.053 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 37872)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+923836872 ms)
2025-02-18 12:33:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923884421 ms)
2025-02-18 12:33:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923884421 ms)
2025-02-18 12:33:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:33:25.402756+00:00', 'runtime': '923880.003', 'adc_vb_f': '12.506', 'adc_va_f': '15.14', 'adc_vl_f': '12.509', 'adc_ic_f': '0.036', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.423', 'sweep_vmp': '15.118', 'sweep_pmax': '5.013', 'sweep_voc': '20.154', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923884422 ms)
2025-02-18 12:33:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:32:25.400605+00:00', 'runtime': '923820.001', 'adc_vb_f': '12.503', 'adc_va_f': '14.987', 'adc_vl_f': '12.509', 'adc_ic_f': '0.019', 'adc_il_f': '0.401', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.211', 'sweep_vmp': '14.963', 'sweep_pmax': '5.013', 'sweep_voc': '19.946', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923884423 ms)
2025-02-18 12:34:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+923944406 ms)
2025-02-18 12:34:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+923944406 ms)
2025-02-18 12:34:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:34:25.400170+00:00', 'runtime': '923940.001', 'adc_vb_f': '12.509', 'adc_va_f': '15.335', 'adc_vl_f': '12.509', 'adc_ic_f': '0.056', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.695', 'sweep_vmp': '15.283', 'sweep_pmax': '5.013', 'sweep_voc': '20.374', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923944410 ms)
2025-02-18 12:34:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:33:25.402756+00:00', 'runtime': '923880.003', 'adc_vb_f': '12.506', 'adc_va_f': '15.14', 'adc_vl_f': '12.509', 'adc_ic_f': '0.036', 'adc_il_f': '0.428', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.423', 'sweep_vmp': '15.118', 'sweep_pmax': '5.013', 'sweep_voc': '20.154', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+923944411 ms)
2025-02-18 12:35:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924004419 ms)
2025-02-18 12:35:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924004419 ms)
2025-02-18 12:35:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:35:25.402793+00:00', 'runtime': '924000.003', 'adc_vb_f': '12.512', 'adc_va_f': '15.472', 'adc_vl_f': '12.509', 'adc_ic_f': '0.085', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.027', 'sweep_vmp': '15.433', 'sweep_pmax': '5.013', 'sweep_voc': '20.575', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924004423 ms)
2025-02-18 12:35:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:34:25.400170+00:00', 'runtime': '923940.001', 'adc_vb_f': '12.509', 'adc_va_f': '15.335', 'adc_vl_f': '12.509', 'adc_ic_f': '0.056', 'adc_il_f': '0.394', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.506', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '0.695', 'sweep_vmp': '15.283', 'sweep_pmax': '5.013', 'sweep_voc': '20.374', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924004424 ms)
2025-02-18 12:35:35.715 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924013534 ms)
2025-02-18 12:35:35.716 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924013535 ms)
2025-02-18 12:35:35.717 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:35:25.400151+00:00'} (T+924013536 ms)
2025-02-18 12:36:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924064418 ms)
2025-02-18 12:36:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924064419 ms)
2025-02-18 12:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:36:25.400157+00:00', 'runtime': '924060.001', 'adc_vb_f': '12.515', 'adc_va_f': '15.524', 'adc_vl_f': '12.509', 'adc_ic_f': '0.106', 'adc_il_f': '0.353', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.329', 'sweep_vmp': '15.521', 'sweep_pmax': '5.013', 'sweep_voc': '20.691', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924064420 ms)
2025-02-18 12:36:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:35:25.402793+00:00', 'runtime': '924000.003', 'adc_vb_f': '12.512', 'adc_va_f': '15.472', 'adc_vl_f': '12.509', 'adc_ic_f': '0.085', 'adc_il_f': '0.384', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.027', 'sweep_vmp': '15.433', 'sweep_pmax': '5.013', 'sweep_voc': '20.575', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924064420 ms)
2025-02-18 12:37:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924124403 ms)
2025-02-18 12:37:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924124404 ms)
2025-02-18 12:37:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:37:25.400036+00:00', 'runtime': '924120.001', 'adc_vb_f': '12.518', 'adc_va_f': '15.625', 'adc_vl_f': '12.509', 'adc_ic_f': '0.118', 'adc_il_f': '0.326', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.449', 'sweep_vmp': '15.604', 'sweep_pmax': '5.013', 'sweep_voc': '20.801', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924124405 ms)
2025-02-18 12:37:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:36:25.400157+00:00', 'runtime': '924060.001', 'adc_vb_f': '12.515', 'adc_va_f': '15.524', 'adc_vl_f': '12.509', 'adc_ic_f': '0.106', 'adc_il_f': '0.353', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.512', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.329', 'sweep_vmp': '15.521', 'sweep_pmax': '5.013', 'sweep_voc': '20.691', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924124405 ms)
2025-02-18 12:38:08.541 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+924166360 ms)
2025-02-18 12:38:08.541 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+924166360 ms)
2025-02-18 12:38:08.542 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38230)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924166361 ms)
2025-02-18 12:38:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924184420 ms)
2025-02-18 12:38:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924184420 ms)
2025-02-18 12:38:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:38:25.400204+00:00', 'runtime': '924180.001', 'adc_vb_f': '12.515', 'adc_va_f': '15.674', 'adc_vl_f': '12.509', 'adc_ic_f': '0.123', 'adc_il_f': '0.365', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.54', 'sweep_vmp': '15.671', 'sweep_pmax': '5.013', 'sweep_voc': '20.892', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924184421 ms)
2025-02-18 12:38:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:37:25.400036+00:00', 'runtime': '924120.001', 'adc_vb_f': '12.518', 'adc_va_f': '15.625', 'adc_vl_f': '12.509', 'adc_ic_f': '0.118', 'adc_il_f': '0.326', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.449', 'sweep_vmp': '15.604', 'sweep_pmax': '5.013', 'sweep_voc': '20.801', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924184422 ms)
2025-02-18 12:38:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924193428 ms)
2025-02-18 12:38:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924193429 ms)
2025-02-18 12:38:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:38:25.413058+00:00'} (T+924193429 ms)
2025-02-18 12:38:48.591 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+924206411 ms)
2025-02-18 12:38:48.592 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+924206411 ms)
2025-02-18 12:38:48.592 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924206412 ms)
2025-02-18 12:38:48.593 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+924206413 ms)
2025-02-18 12:38:48.594 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+924206413 ms)
2025-02-18 12:38:48.594 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924206413 ms)
2025-02-18 12:38:48.595 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+924206414 ms)
2025-02-18 12:38:48.595 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+924206415 ms)
2025-02-18 12:38:48.596 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924206415 ms)
2025-02-18 12:38:48.597 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+924206416 ms)
2025-02-18 12:38:48.597 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+924206417 ms)
2025-02-18 12:38:48.598 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924206417 ms)
2025-02-18 12:38:48.599 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+924206418 ms)
2025-02-18 12:38:48.599 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+924206419 ms)
2025-02-18 12:38:48.600 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924206419 ms)
2025-02-18 12:39:08.691 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+924226510 ms)
2025-02-18 12:39:08.691 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+924226511 ms)
2025-02-18 12:39:08.692 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38296)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924226511 ms)
2025-02-18 12:39:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924244419 ms)
2025-02-18 12:39:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924244420 ms)
2025-02-18 12:39:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:39:25.400158+00:00', 'runtime': '924240.001', 'adc_vb_f': '12.518', 'adc_va_f': '15.723', 'adc_vl_f': '12.509', 'adc_ic_f': '0.126', 'adc_il_f': '0.343', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.57', 'sweep_vmp': '15.686', 'sweep_pmax': '5.013', 'sweep_voc': '20.911', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924244421 ms)
2025-02-18 12:39:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:38:25.400204+00:00', 'runtime': '924180.001', 'adc_vb_f': '12.515', 'adc_va_f': '15.674', 'adc_vl_f': '12.509', 'adc_ic_f': '0.123', 'adc_il_f': '0.365', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.515', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.54', 'sweep_vmp': '15.671', 'sweep_pmax': '5.013', 'sweep_voc': '20.892', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924244421 ms)
2025-02-18 12:39:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924253422 ms)
2025-02-18 12:39:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924253422 ms)
2025-02-18 12:39:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:39:25.405134+00:00'} (T+924253423 ms)
2025-02-18 12:40:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924304417 ms)
2025-02-18 12:40:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924304417 ms)
2025-02-18 12:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:40:25.402887+00:00', 'runtime': '924300.003', 'adc_vb_f': '12.521', 'adc_va_f': '15.726', 'adc_vl_f': '12.509', 'adc_ic_f': '0.133', 'adc_il_f': '0.324', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.661', 'sweep_vmp': '15.686', 'sweep_pmax': '5.013', 'sweep_voc': '20.911', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924304419 ms)
2025-02-18 12:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:39:25.400158+00:00', 'runtime': '924240.001', 'adc_vb_f': '12.518', 'adc_va_f': '15.723', 'adc_vl_f': '12.509', 'adc_ic_f': '0.126', 'adc_il_f': '0.343', 't_hs': '3', 't_batt': '4', 't_amb': '4', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '14.443', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.496', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.57', 'sweep_vmp': '15.686', 'sweep_pmax': '5.013', 'sweep_voc': '20.911', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924304420 ms)
2025-02-18 12:41:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924364419 ms)
2025-02-18 12:41:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924364419 ms)
2025-02-18 12:41:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:41:25.401083+00:00', 'runtime': '924360.002', 'adc_vb_f': '12.518', 'adc_va_f': '15.723', 'adc_vl_f': '12.509', 'adc_ic_f': '0.14', 'adc_il_f': '0.326', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.751', 'sweep_vmp': '15.704', 'sweep_pmax': '5.013', 'sweep_voc': '20.938', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924364420 ms)
2025-02-18 12:41:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:40:25.402887+00:00', 'runtime': '924300.003', 'adc_vb_f': '12.521', 'adc_va_f': '15.726', 'adc_vl_f': '12.509', 'adc_ic_f': '0.133', 'adc_il_f': '0.324', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.661', 'sweep_vmp': '15.686', 'sweep_pmax': '5.013', 'sweep_voc': '20.911', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924364421 ms)
2025-02-18 12:42:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924424416 ms)
2025-02-18 12:42:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924424417 ms)
2025-02-18 12:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:42:25.400169+00:00', 'runtime': '924420.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.765', 'adc_vl_f': '12.509', 'adc_ic_f': '0.152', 'adc_il_f': '0.372', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.872', 'sweep_vmp': '15.768', 'sweep_pmax': '5.013', 'sweep_voc': '21.021', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924424418 ms)
2025-02-18 12:42:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:41:25.401083+00:00', 'runtime': '924360.002', 'adc_vb_f': '12.518', 'adc_va_f': '15.723', 'adc_vl_f': '12.509', 'adc_ic_f': '0.14', 'adc_il_f': '0.326', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.518', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.751', 'sweep_vmp': '15.704', 'sweep_pmax': '5.013', 'sweep_voc': '20.938', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924424418 ms)
2025-02-18 12:42:31.013 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+924428832 ms)
2025-02-18 12:42:31.013 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+924428832 ms)
2025-02-18 12:42:31.014 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38510)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924428833 ms)
2025-02-18 12:42:36.122 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924433942 ms)
2025-02-18 12:42:36.123 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924433942 ms)
2025-02-18 12:42:36.124 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:42:25.401089+00:00'} (T+924433944 ms)
2025-02-18 12:43:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924484416 ms)
2025-02-18 12:43:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924484416 ms)
2025-02-18 12:43:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:43:25.400190+00:00', 'runtime': '924480.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.817', 'adc_vl_f': '12.509', 'adc_ic_f': '0.159', 'adc_il_f': '0.37', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.993', 'sweep_vmp': '15.775', 'sweep_pmax': '5.013', 'sweep_voc': '21.03', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924484419 ms)
2025-02-18 12:43:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:42:25.400169+00:00', 'runtime': '924420.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.765', 'adc_vl_f': '12.509', 'adc_ic_f': '0.152', 'adc_il_f': '0.372', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.872', 'sweep_vmp': '15.768', 'sweep_pmax': '5.013', 'sweep_voc': '21.021', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924484419 ms)
2025-02-18 12:44:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924544415 ms)
2025-02-18 12:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924544415 ms)
2025-02-18 12:44:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:44:25.399998+00:00', 'runtime': '924540.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.872', 'adc_vl_f': '12.509', 'adc_ic_f': '0.169', 'adc_il_f': '0.353', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.114', 'sweep_vmp': '15.842', 'sweep_pmax': '5.013', 'sweep_voc': '21.118', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924544417 ms)
2025-02-18 12:44:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:43:25.400190+00:00', 'runtime': '924480.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.817', 'adc_vl_f': '12.509', 'adc_ic_f': '0.159', 'adc_il_f': '0.37', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '1.993', 'sweep_vmp': '15.775', 'sweep_pmax': '5.013', 'sweep_voc': '21.03', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924544417 ms)
2025-02-18 12:45:24.304 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+924602123 ms)
2025-02-18 12:45:24.304 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+924602124 ms)
2025-02-18 12:45:24.305 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38694)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924602124 ms)
2025-02-18 12:45:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924604415 ms)
2025-02-18 12:45:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924604415 ms)
2025-02-18 12:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:45:25.400154+00:00', 'runtime': '924600.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.872', 'adc_vl_f': '12.509', 'adc_ic_f': '0.184', 'adc_il_f': '0.345', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.265', 'sweep_vmp': '15.842', 'sweep_pmax': '5.013', 'sweep_voc': '21.118', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924604416 ms)
2025-02-18 12:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:44:25.399998+00:00', 'runtime': '924540.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.872', 'adc_vl_f': '12.509', 'adc_ic_f': '0.169', 'adc_il_f': '0.353', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.114', 'sweep_vmp': '15.842', 'sweep_pmax': '5.013', 'sweep_voc': '21.118', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924604417 ms)
2025-02-18 12:45:28.384 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+924606203 ms)
2025-02-18 12:45:28.384 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+924606204 ms)
2025-02-18 12:45:28.385 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38700)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924606204 ms)
2025-02-18 12:46:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924664415 ms)
2025-02-18 12:46:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924664415 ms)
2025-02-18 12:46:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:46:25.400152+00:00', 'runtime': '924660.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.961', 'adc_vl_f': '12.509', 'adc_ic_f': '0.193', 'adc_il_f': '0.319', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.416', 'sweep_vmp': '15.924', 'sweep_pmax': '5.013', 'sweep_voc': '21.228', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924664416 ms)
2025-02-18 12:46:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:45:25.400154+00:00', 'runtime': '924600.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.872', 'adc_vl_f': '12.509', 'adc_ic_f': '0.184', 'adc_il_f': '0.345', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.265', 'sweep_vmp': '15.842', 'sweep_pmax': '5.013', 'sweep_voc': '21.118', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924664417 ms)
2025-02-18 12:46:35.611 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924673430 ms)
2025-02-18 12:46:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924673430 ms)
2025-02-18 12:46:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:46:25.400327+00:00'} (T+924673431 ms)
2025-02-18 12:47:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924724416 ms)
2025-02-18 12:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924724417 ms)
2025-02-18 12:47:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:47:25.400157+00:00', 'runtime': '924720.001', 'adc_vb_f': '12.534', 'adc_va_f': '15.961', 'adc_vl_f': '12.509', 'adc_ic_f': '0.205', 'adc_il_f': '0.333', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.567', 'sweep_vmp': '15.939', 'sweep_pmax': '5.013', 'sweep_voc': '21.249', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924724418 ms)
2025-02-18 12:47:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:46:25.400152+00:00', 'runtime': '924660.001', 'adc_vb_f': '12.521', 'adc_va_f': '15.961', 'adc_vl_f': '12.509', 'adc_ic_f': '0.193', 'adc_il_f': '0.319', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.521', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.416', 'sweep_vmp': '15.924', 'sweep_pmax': '5.013', 'sweep_voc': '21.228', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924724418 ms)
2025-02-18 12:47:30.497 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: connect to host hamma.dev port 22: Network is unreachable
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
 (T+924728316 ms)
2025-02-18 12:48:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924784415 ms)
2025-02-18 12:48:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924784416 ms)
2025-02-18 12:48:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:48:25.400179+00:00', 'runtime': '924780.001', 'adc_vb_f': '12.552', 'adc_va_f': '16.013', 'adc_vl_f': '12.509', 'adc_ic_f': '0.222', 'adc_il_f': '0.321', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.748', 'sweep_vmp': '16.006', 'sweep_pmax': '5.013', 'sweep_voc': '21.338', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924784418 ms)
2025-02-18 12:48:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:47:25.400157+00:00', 'runtime': '924720.001', 'adc_vb_f': '12.534', 'adc_va_f': '15.961', 'adc_vl_f': '12.509', 'adc_ic_f': '0.205', 'adc_il_f': '0.333', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.527', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.7', 'ahl_t': '1598.7', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.567', 'sweep_vmp': '15.939', 'sweep_pmax': '5.013', 'sweep_voc': '21.249', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924784418 ms)
2025-02-18 12:49:02.725 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+924820544 ms)
2025-02-18 12:49:02.725 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+924820544 ms)
2025-02-18 12:49:02.726 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 38936)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+924820545 ms)
2025-02-18 12:49:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924844416 ms)
2025-02-18 12:49:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924844417 ms)
2025-02-18 12:49:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:49:25.400185+00:00', 'runtime': '924840.001', 'adc_vb_f': '12.555', 'adc_va_f': '16.116', 'adc_vl_f': '12.509', 'adc_ic_f': '0.237', 'adc_il_f': '0.348', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.959', 'sweep_vmp': '16.089', 'sweep_pmax': '5.013', 'sweep_voc': '21.448', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924844418 ms)
2025-02-18 12:49:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:48:25.400179+00:00', 'runtime': '924780.001', 'adc_vb_f': '12.552', 'adc_va_f': '16.013', 'adc_vl_f': '12.509', 'adc_ic_f': '0.222', 'adc_il_f': '0.321', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.54', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.748', 'sweep_vmp': '16.006', 'sweep_pmax': '5.013', 'sweep_voc': '21.338', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924844418 ms)
2025-02-18 12:49:36.080 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924853900 ms)
2025-02-18 12:49:36.081 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+924853900 ms)
2025-02-18 12:49:36.082 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 12:49:25.400346+00:00'} (T+924853901 ms)
2025-02-18 12:50:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924904412 ms)
2025-02-18 12:50:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924904412 ms)
2025-02-18 12:50:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:50:25.402828+00:00', 'runtime': '924900.003', 'adc_vb_f': '12.558', 'adc_va_f': '16.214', 'adc_vl_f': '12.509', 'adc_ic_f': '0.254', 'adc_il_f': '0.36', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.171', 'sweep_vmp': '16.171', 'sweep_pmax': '5.013', 'sweep_voc': '21.558', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924904416 ms)
2025-02-18 12:50:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:49:25.400185+00:00', 'runtime': '924840.001', 'adc_vb_f': '12.555', 'adc_va_f': '16.116', 'adc_vl_f': '12.509', 'adc_ic_f': '0.237', 'adc_il_f': '0.348', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.552', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '2.959', 'sweep_vmp': '16.089', 'sweep_pmax': '5.013', 'sweep_voc': '21.448', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924904416 ms)
2025-02-18 12:51:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+924964405 ms)
2025-02-18 12:51:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+924964405 ms)
2025-02-18 12:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:51:25.400799+00:00', 'runtime': '924960.001', 'adc_vb_f': '12.558', 'adc_va_f': '16.751', 'adc_vl_f': '12.509', 'adc_ic_f': '0.273', 'adc_il_f': '0.367', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.427', 'sweep_vmp': '16.788', 'sweep_pmax': '3.911', 'sweep_voc': '21.609', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924964409 ms)
2025-02-18 12:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:50:25.402828+00:00', 'runtime': '924900.003', 'adc_vb_f': '12.558', 'adc_va_f': '16.214', 'adc_vl_f': '12.509', 'adc_ic_f': '0.254', 'adc_il_f': '0.36', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.555', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.171', 'sweep_vmp': '16.171', 'sweep_pmax': '5.013', 'sweep_voc': '21.558', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+924964410 ms)
2025-02-18 12:52:20.064 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925017883 ms)
2025-02-18 12:52:20.064 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925017884 ms)
2025-02-18 12:52:20.065 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39144)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925017884 ms)
2025-02-18 12:52:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925024416 ms)
2025-02-18 12:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925024417 ms)
2025-02-18 12:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:52:25.400646+00:00', 'runtime': '925020.001', 'adc_vb_f': '12.558', 'adc_va_f': '16.751', 'adc_vl_f': '12.509', 'adc_ic_f': '0.287', 'adc_il_f': '0.37', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.609', 'sweep_vmp': '16.788', 'sweep_pmax': '3.911', 'sweep_voc': '21.609', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925024418 ms)
2025-02-18 12:52:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:51:25.400799+00:00', 'runtime': '924960.001', 'adc_vb_f': '12.558', 'adc_va_f': '16.751', 'adc_vl_f': '12.509', 'adc_ic_f': '0.273', 'adc_il_f': '0.367', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.427', 'sweep_vmp': '16.788', 'sweep_pmax': '3.911', 'sweep_voc': '21.609', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925024418 ms)
2025-02-18 12:53:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925084414 ms)
2025-02-18 12:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925084414 ms)
2025-02-18 12:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:53:25.400184+00:00', 'runtime': '925080.001', 'adc_vb_f': '12.558', 'adc_va_f': '16.751', 'adc_vl_f': '12.527', 'adc_ic_f': '0.309', 'adc_il_f': '0.367', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.85', 'sweep_vmp': '16.788', 'sweep_pmax': '3.911', 'sweep_voc': '21.609', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925084415 ms)
2025-02-18 12:53:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:52:25.400646+00:00', 'runtime': '925020.001', 'adc_vb_f': '12.558', 'adc_va_f': '16.751', 'adc_vl_f': '12.509', 'adc_ic_f': '0.287', 'adc_il_f': '0.37', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.609', 'sweep_vmp': '16.788', 'sweep_pmax': '3.911', 'sweep_voc': '21.609', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925084416 ms)
2025-02-18 12:54:25.641 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925143460 ms)
2025-02-18 12:54:25.641 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925143461 ms)
2025-02-18 12:54:25.642 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:54:25.400787+00:00', 'runtime': '925140.001', 'adc_vb_f': '12.585', 'adc_va_f': '19.528', 'adc_vl_f': '12.54', 'adc_ic_f': '0.362', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.567', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.545', 'sweep_vmp': '19.534', 'sweep_pmax': '4.62', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925143462 ms)
2025-02-18 12:54:25.643 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:53:25.400184+00:00', 'runtime': '925080.001', 'adc_vb_f': '12.558', 'adc_va_f': '16.751', 'adc_vl_f': '12.527', 'adc_ic_f': '0.309', 'adc_il_f': '0.367', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.558', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20143', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '3.85', 'sweep_vmp': '16.788', 'sweep_pmax': '3.911', 'sweep_voc': '21.609', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925143462 ms)
2025-02-18 12:54:58.336 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925176155 ms)
2025-02-18 12:54:58.336 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925176155 ms)
2025-02-18 12:54:58.336 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39312)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925176156 ms)
2025-02-18 12:55:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925204410 ms)
2025-02-18 12:55:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925204411 ms)
2025-02-18 12:55:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:55:25.400515+00:00', 'runtime': '925200.001', 'adc_vb_f': '12.589', 'adc_va_f': '19.431', 'adc_vl_f': '12.546', 'adc_ic_f': '0.37', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.635', 'sweep_vmp': '19.479', 'sweep_pmax': '4.771', 'sweep_voc': '21.811', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925204412 ms)
2025-02-18 12:55:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:54:25.400787+00:00', 'runtime': '925140.001', 'adc_vb_f': '12.585', 'adc_va_f': '19.528', 'adc_vl_f': '12.54', 'adc_ic_f': '0.362', 'adc_il_f': '0.387', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.567', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.545', 'sweep_vmp': '19.534', 'sweep_pmax': '4.62', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925204412 ms)
2025-02-18 12:56:08.425 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+925246245 ms)
2025-02-18 12:56:08.426 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+925246245 ms)
2025-02-18 12:56:08.426 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925246246 ms)
2025-02-18 12:56:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925264411 ms)
2025-02-18 12:56:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925264411 ms)
2025-02-18 12:56:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:56:25.402726+00:00', 'runtime': '925260.003', 'adc_vb_f': '12.589', 'adc_va_f': '19.434', 'adc_vl_f': '12.546', 'adc_ic_f': '0.37', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.62', 'sweep_vmp': '19.479', 'sweep_pmax': '4.771', 'sweep_voc': '21.811', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925264412 ms)
2025-02-18 12:56:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:55:25.400515+00:00', 'runtime': '925200.001', 'adc_vb_f': '12.589', 'adc_va_f': '19.431', 'adc_vl_f': '12.546', 'adc_ic_f': '0.37', 'adc_il_f': '0.413', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.585', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.635', 'sweep_vmp': '19.479', 'sweep_pmax': '4.771', 'sweep_voc': '21.811', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925264413 ms)
2025-02-18 12:57:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925324408 ms)
2025-02-18 12:57:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925324408 ms)
2025-02-18 12:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:57:25.400179+00:00', 'runtime': '925320.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.431', 'adc_vl_f': '12.546', 'adc_ic_f': '0.365', 'adc_il_f': '0.36', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.59', 'sweep_vmp': '19.479', 'sweep_pmax': '4.771', 'sweep_voc': '21.811', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925324409 ms)
2025-02-18 12:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:56:25.402726+00:00', 'runtime': '925260.003', 'adc_vb_f': '12.589', 'adc_va_f': '19.434', 'adc_vl_f': '12.546', 'adc_ic_f': '0.37', 'adc_il_f': '0.425', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.592', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.62', 'sweep_vmp': '19.479', 'sweep_pmax': '4.771', 'sweep_voc': '21.811', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925324410 ms)
2025-02-18 12:58:08.626 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925366445 ms)
2025-02-18 12:58:08.626 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925366445 ms)
2025-02-18 12:58:08.627 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39522)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925366446 ms)
2025-02-18 12:58:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925384417 ms)
2025-02-18 12:58:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925384418 ms)
2025-02-18 12:58:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:58:25.400183+00:00', 'runtime': '925380.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.546', 'adc_ic_f': '0.377', 'adc_il_f': '0.341', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.711', 'sweep_vmp': '19.281', 'sweep_pmax': '4.832', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925384419 ms)
2025-02-18 12:58:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:57:25.400179+00:00', 'runtime': '925320.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.431', 'adc_vl_f': '12.546', 'adc_ic_f': '0.365', 'adc_il_f': '0.36', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.59', 'sweep_vmp': '19.479', 'sweep_pmax': '4.771', 'sweep_voc': '21.811', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925384419 ms)
2025-02-18 12:59:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925444415 ms)
2025-02-18 12:59:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925444415 ms)
2025-02-18 12:59:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 12:59:25.400189+00:00', 'runtime': '925440.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.546', 'adc_ic_f': '0.387', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.847', 'sweep_vmp': '19.281', 'sweep_pmax': '4.832', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925444416 ms)
2025-02-18 12:59:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:58:25.400183+00:00', 'runtime': '925380.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.546', 'adc_ic_f': '0.377', 'adc_il_f': '0.341', 't_hs': '3', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.711', 'sweep_vmp': '19.281', 'sweep_pmax': '4.832', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925444417 ms)
2025-02-18 13:00:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925504410 ms)
2025-02-18 13:00:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925504410 ms)
2025-02-18 13:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:00:25.400192+00:00', 'runtime': '925500.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.546', 'adc_ic_f': '0.399', 'adc_il_f': '0.374', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.013', 'sweep_vmp': '19.281', 'sweep_pmax': '4.832', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925504411 ms)
2025-02-18 13:00:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 12:59:25.400189+00:00', 'runtime': '925440.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.546', 'adc_ic_f': '0.387', 'adc_il_f': '0.411', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '4.847', 'sweep_vmp': '19.281', 'sweep_pmax': '4.832', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925504412 ms)
2025-02-18 13:00:32.870 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925510689 ms)
2025-02-18 13:00:32.870 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925510690 ms)
2025-02-18 13:00:32.871 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39674)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925510690 ms)
2025-02-18 13:00:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925513428 ms)
2025-02-18 13:00:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925513429 ms)
2025-02-18 13:00:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:00:25.400347+00:00'} (T+925513430 ms)
2025-02-18 13:01:06.981 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925544801 ms)
2025-02-18 13:01:06.982 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925544801 ms)
2025-02-18 13:01:06.982 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39714)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925544802 ms)
2025-02-18 13:01:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925564406 ms)
2025-02-18 13:01:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925564406 ms)
2025-02-18 13:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:01:25.400264+00:00', 'runtime': '925560.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.241', 'adc_vl_f': '12.549', 'adc_ic_f': '0.411', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.164', 'sweep_vmp': '19.284', 'sweep_pmax': '5.224', 'sweep_voc': '21.786', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925564407 ms)
2025-02-18 13:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:00:25.400192+00:00', 'runtime': '925500.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.546', 'adc_ic_f': '0.399', 'adc_il_f': '0.374', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.013', 'sweep_vmp': '19.281', 'sweep_pmax': '4.832', 'sweep_voc': '21.765', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925564408 ms)
2025-02-18 13:02:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925624406 ms)
2025-02-18 13:02:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925624407 ms)
2025-02-18 13:02:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:02:25.400214+00:00', 'runtime': '925620.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.579', 'adc_ic_f': '0.423', 'adc_il_f': '0.374', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.285', 'sweep_vmp': '19.284', 'sweep_pmax': '5.224', 'sweep_voc': '21.786', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925624408 ms)
2025-02-18 13:02:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:01:25.400264+00:00', 'runtime': '925560.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.241', 'adc_vl_f': '12.549', 'adc_ic_f': '0.411', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.164', 'sweep_vmp': '19.284', 'sweep_pmax': '5.224', 'sweep_voc': '21.786', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925624408 ms)
2025-02-18 13:03:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925684420 ms)
2025-02-18 13:03:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925684421 ms)
2025-02-18 13:03:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:03:25.400171+00:00', 'runtime': '925680.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.241', 'adc_vl_f': '12.582', 'adc_ic_f': '0.437', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.496', 'sweep_vmp': '19.284', 'sweep_pmax': '5.224', 'sweep_voc': '21.786', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925684422 ms)
2025-02-18 13:03:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:02:25.400214+00:00', 'runtime': '925620.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.244', 'adc_vl_f': '12.579', 'adc_ic_f': '0.423', 'adc_il_f': '0.374', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.285', 'sweep_vmp': '19.284', 'sweep_pmax': '5.224', 'sweep_voc': '21.786', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925684422 ms)
2025-02-18 13:03:28.256 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925686076 ms)
2025-02-18 13:03:28.257 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925686076 ms)
2025-02-18 13:03:28.257 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39862)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925686077 ms)
2025-02-18 13:03:35.592 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925693412 ms)
2025-02-18 13:03:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925693412 ms)
2025-02-18 13:03:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:03:25.400327+00:00'} (T+925693413 ms)
2025-02-18 13:03:48.327 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925706147 ms)
2025-02-18 13:03:48.328 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925706147 ms)
2025-02-18 13:03:48.328 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39888)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925706148 ms)
2025-02-18 13:04:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925744416 ms)
2025-02-18 13:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925744416 ms)
2025-02-18 13:04:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:04:25.400161+00:00', 'runtime': '925740.001', 'adc_vb_f': '12.601', 'adc_va_f': '19.434', 'adc_vl_f': '12.585', 'adc_ic_f': '0.447', 'adc_il_f': '0.338', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.617', 'sweep_vmp': '19.446', 'sweep_pmax': '5.692', 'sweep_voc': '21.875', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925744417 ms)
2025-02-18 13:04:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:03:25.400171+00:00', 'runtime': '925680.001', 'adc_vb_f': '12.598', 'adc_va_f': '19.241', 'adc_vl_f': '12.582', 'adc_ic_f': '0.437', 'adc_il_f': '0.394', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.598', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.496', 'sweep_vmp': '19.284', 'sweep_pmax': '5.224', 'sweep_voc': '21.786', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925744418 ms)
2025-02-18 13:04:35.591 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925753410 ms)
2025-02-18 13:04:35.592 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925753411 ms)
2025-02-18 13:04:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:04:25.400218+00:00'} (T+925753412 ms)
2025-02-18 13:04:51.465 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925769284 ms)
2025-02-18 13:04:51.465 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925769285 ms)
2025-02-18 13:04:51.466 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 39956)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925769286 ms)
2025-02-18 13:05:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925804415 ms)
2025-02-18 13:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925804415 ms)
2025-02-18 13:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:05:25.400163+00:00', 'runtime': '925800.001', 'adc_vb_f': '12.604', 'adc_va_f': '19.434', 'adc_vl_f': '12.585', 'adc_ic_f': '0.454', 'adc_il_f': '0.348', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.722', 'sweep_vmp': '19.446', 'sweep_pmax': '5.692', 'sweep_voc': '21.875', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925804416 ms)
2025-02-18 13:05:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:04:25.400161+00:00', 'runtime': '925740.001', 'adc_vb_f': '12.601', 'adc_va_f': '19.434', 'adc_vl_f': '12.585', 'adc_ic_f': '0.447', 'adc_il_f': '0.338', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.595', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.8', 'ahl_t': '1598.8', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.617', 'sweep_vmp': '19.446', 'sweep_pmax': '5.692', 'sweep_voc': '21.875', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925804417 ms)
2025-02-18 13:05:35.688 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925813507 ms)
2025-02-18 13:05:35.688 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925813508 ms)
2025-02-18 13:05:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:05:25.400213+00:00'} (T+925813509 ms)
2025-02-18 13:05:52.616 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925830435 ms)
2025-02-18 13:05:52.616 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925830435 ms)
2025-02-18 13:05:52.617 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40032)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925830436 ms)
2025-02-18 13:06:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925864414 ms)
2025-02-18 13:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925864415 ms)
2025-02-18 13:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:06:25.400146+00:00', 'runtime': '925860.001', 'adc_vb_f': '12.607', 'adc_va_f': '19.431', 'adc_vl_f': '12.585', 'adc_ic_f': '0.464', 'adc_il_f': '0.358', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.604', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.783', 'sweep_vmp': '19.446', 'sweep_pmax': '5.692', 'sweep_voc': '21.875', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925864416 ms)
2025-02-18 13:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:05:25.400163+00:00', 'runtime': '925800.001', 'adc_vb_f': '12.604', 'adc_va_f': '19.434', 'adc_vl_f': '12.585', 'adc_ic_f': '0.454', 'adc_il_f': '0.348', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.601', 'vb_ref': '14.414', 'ahc_r': '2660.4', 'ahc_t': '2660.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.722', 'sweep_vmp': '19.446', 'sweep_pmax': '5.692', 'sweep_voc': '21.875', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.0', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925864416 ms)
2025-02-18 13:07:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925924413 ms)
2025-02-18 13:07:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925924413 ms)
2025-02-18 13:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:07:25.400152+00:00', 'runtime': '925920.001', 'adc_vb_f': '12.607', 'adc_va_f': '18.802', 'adc_vl_f': '12.585', 'adc_ic_f': '0.476', 'adc_il_f': '0.333', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.607', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.934', 'sweep_vmp': '18.845', 'sweep_pmax': '6.085', 'sweep_voc': '21.924', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925924414 ms)
2025-02-18 13:07:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:06:25.400146+00:00', 'runtime': '925860.001', 'adc_vb_f': '12.607', 'adc_va_f': '19.431', 'adc_vl_f': '12.585', 'adc_ic_f': '0.464', 'adc_il_f': '0.358', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.604', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.783', 'sweep_vmp': '19.446', 'sweep_pmax': '5.692', 'sweep_voc': '21.875', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925924414 ms)
2025-02-18 13:08:13.886 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+925971705 ms)
2025-02-18 13:08:13.886 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+925971705 ms)
2025-02-18 13:08:13.887 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40180)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+925971706 ms)
2025-02-18 13:08:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+925984411 ms)
2025-02-18 13:08:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+925984411 ms)
2025-02-18 13:08:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:08:25.402755+00:00', 'runtime': '925980.003', 'adc_vb_f': '12.616', 'adc_va_f': '18.802', 'adc_vl_f': '12.585', 'adc_ic_f': '0.49', 'adc_il_f': '0.379', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.175', 'sweep_vmp': '18.845', 'sweep_pmax': '6.085', 'sweep_voc': '21.924', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925984412 ms)
2025-02-18 13:08:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:07:25.400152+00:00', 'runtime': '925920.001', 'adc_vb_f': '12.607', 'adc_va_f': '18.802', 'adc_vl_f': '12.585', 'adc_ic_f': '0.476', 'adc_il_f': '0.333', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.607', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '5.934', 'sweep_vmp': '18.845', 'sweep_pmax': '6.085', 'sweep_voc': '21.924', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+925984413 ms)
2025-02-18 13:08:35.600 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925993419 ms)
2025-02-18 13:08:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+925993420 ms)
2025-02-18 13:08:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:08:25.400122+00:00'} (T+925993421 ms)
2025-02-18 13:09:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926044411 ms)
2025-02-18 13:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926044412 ms)
2025-02-18 13:09:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:09:25.400522+00:00', 'runtime': '926040.001', 'adc_vb_f': '12.625', 'adc_va_f': '18.799', 'adc_vl_f': '12.585', 'adc_ic_f': '0.507', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.402', 'sweep_vmp': '18.845', 'sweep_pmax': '6.085', 'sweep_voc': '21.924', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926044413 ms)
2025-02-18 13:09:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:08:25.402755+00:00', 'runtime': '925980.003', 'adc_vb_f': '12.616', 'adc_va_f': '18.802', 'adc_vl_f': '12.585', 'adc_ic_f': '0.49', 'adc_il_f': '0.379', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.613', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.175', 'sweep_vmp': '18.845', 'sweep_pmax': '6.085', 'sweep_voc': '21.924', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926044415 ms)
2025-02-18 13:10:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926104407 ms)
2025-02-18 13:10:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926104408 ms)
2025-02-18 13:10:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:10:25.400174+00:00', 'runtime': '926100.001', 'adc_vb_f': '12.634', 'adc_va_f': '18.896', 'adc_vl_f': '12.601', 'adc_ic_f': '0.524', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.583', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '21.985', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926104409 ms)
2025-02-18 13:10:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:09:25.400522+00:00', 'runtime': '926040.001', 'adc_vb_f': '12.625', 'adc_va_f': '18.799', 'adc_vl_f': '12.585', 'adc_ic_f': '0.507', 'adc_il_f': '0.418', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.625', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.402', 'sweep_vmp': '18.845', 'sweep_pmax': '6.085', 'sweep_voc': '21.924', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926104412 ms)
2025-02-18 13:11:08.167 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+926145986 ms)
2025-02-18 13:11:08.167 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+926145987 ms)
2025-02-18 13:11:08.168 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40366)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+926145987 ms)
2025-02-18 13:11:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926164406 ms)
2025-02-18 13:11:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926164407 ms)
2025-02-18 13:11:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:11:25.400691+00:00', 'runtime': '926160.001', 'adc_vb_f': '12.634', 'adc_va_f': '18.896', 'adc_vl_f': '12.619', 'adc_ic_f': '0.536', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.734', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '21.985', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926164408 ms)
2025-02-18 13:11:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:10:25.400174+00:00', 'runtime': '926100.001', 'adc_vb_f': '12.634', 'adc_va_f': '18.896', 'adc_vl_f': '12.601', 'adc_ic_f': '0.524', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.631', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.583', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '21.985', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926164408 ms)
2025-02-18 13:12:26.602 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926224421 ms)
2025-02-18 13:12:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926224421 ms)
2025-02-18 13:12:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:12:25.402758+00:00', 'runtime': '926220.003', 'adc_vb_f': '12.634', 'adc_va_f': '18.896', 'adc_vl_f': '12.622', 'adc_ic_f': '0.548', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.885', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '21.985', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926224422 ms)
2025-02-18 13:12:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:11:25.400691+00:00', 'runtime': '926160.001', 'adc_vb_f': '12.634', 'adc_va_f': '18.896', 'adc_vl_f': '12.619', 'adc_ic_f': '0.536', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.734', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '21.985', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926224423 ms)
2025-02-18 13:12:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926233424 ms)
2025-02-18 13:12:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926233424 ms)
2025-02-18 13:12:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:12:25.400138+00:00'} (T+926233425 ms)
2025-02-18 13:13:17.406 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+926275225 ms)
2025-02-18 13:13:17.406 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+926275225 ms)
2025-02-18 13:13:17.407 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 40502)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+926275226 ms)
2025-02-18 13:13:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926284419 ms)
2025-02-18 13:13:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926284419 ms)
2025-02-18 13:13:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:13:25.400181+00:00', 'runtime': '926280.001', 'adc_vb_f': '12.643', 'adc_va_f': '18.893', 'adc_vl_f': '12.622', 'adc_ic_f': '0.56', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.036', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926284420 ms)
2025-02-18 13:13:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:12:25.402758+00:00', 'runtime': '926220.003', 'adc_vb_f': '12.634', 'adc_va_f': '18.896', 'adc_vl_f': '12.622', 'adc_ic_f': '0.548', 'adc_il_f': '0.401', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '6.885', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '21.985', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926284421 ms)
2025-02-18 13:14:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926344404 ms)
2025-02-18 13:14:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926344404 ms)
2025-02-18 13:14:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:14:25.400158+00:00', 'runtime': '926340.001', 'adc_vb_f': '12.662', 'adc_va_f': '19.336', 'adc_vl_f': '12.622', 'adc_ic_f': '0.575', 'adc_il_f': '0.358', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.278', 'sweep_vmp': '19.37', 'sweep_pmax': '7.232', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926344405 ms)
2025-02-18 13:14:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:13:25.400181+00:00', 'runtime': '926280.001', 'adc_vb_f': '12.643', 'adc_va_f': '18.893', 'adc_vl_f': '12.622', 'adc_ic_f': '0.56', 'adc_il_f': '0.42', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.634', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.036', 'sweep_vmp': '18.912', 'sweep_pmax': '6.734', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926344406 ms)
2025-02-18 13:15:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926404405 ms)
2025-02-18 13:15:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926404405 ms)
2025-02-18 13:15:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:15:25.400169+00:00', 'runtime': '926400.001', 'adc_vb_f': '12.671', 'adc_va_f': '19.339', 'adc_vl_f': '12.622', 'adc_ic_f': '0.592', 'adc_il_f': '0.365', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.459', 'sweep_vmp': '19.37', 'sweep_pmax': '7.232', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926404406 ms)
2025-02-18 13:15:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:14:25.400158+00:00', 'runtime': '926340.001', 'adc_vb_f': '12.662', 'adc_va_f': '19.336', 'adc_vl_f': '12.622', 'adc_ic_f': '0.575', 'adc_il_f': '0.358', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.65', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.278', 'sweep_vmp': '19.37', 'sweep_pmax': '7.232', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926404407 ms)
2025-02-18 13:16:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926464421 ms)
2025-02-18 13:16:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926464421 ms)
2025-02-18 13:16:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:16:25.402802+00:00', 'runtime': '926460.003', 'adc_vb_f': '12.674', 'adc_va_f': '19.339', 'adc_vl_f': '12.631', 'adc_ic_f': '0.609', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.671', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.746', 'sweep_vmp': '19.37', 'sweep_pmax': '7.232', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926464422 ms)
2025-02-18 13:16:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:15:25.400169+00:00', 'runtime': '926400.001', 'adc_vb_f': '12.671', 'adc_va_f': '19.339', 'adc_vl_f': '12.622', 'adc_ic_f': '0.592', 'adc_il_f': '0.365', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.662', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.459', 'sweep_vmp': '19.37', 'sweep_pmax': '7.232', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926464423 ms)
2025-02-18 13:17:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926524404 ms)
2025-02-18 13:17:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926524404 ms)
2025-02-18 13:17:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:17:25.400582+00:00', 'runtime': '926520.001', 'adc_vb_f': '12.674', 'adc_va_f': '19.196', 'adc_vl_f': '12.659', 'adc_ic_f': '0.626', 'adc_il_f': '0.343', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '14.414', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.957', 'sweep_vmp': '19.196', 'sweep_pmax': '7.881', 'sweep_voc': '22.095', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926524405 ms)
2025-02-18 13:17:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:16:25.402802+00:00', 'runtime': '926460.003', 'adc_vb_f': '12.674', 'adc_va_f': '19.339', 'adc_vl_f': '12.631', 'adc_ic_f': '0.609', 'adc_il_f': '0.382', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.671', 'vb_ref': '14.414', 'ahc_r': '2660.5', 'ahc_t': '2660.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.746', 'sweep_vmp': '19.37', 'sweep_pmax': '7.232', 'sweep_voc': '22.076', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.1', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926524406 ms)
2025-02-18 13:18:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926584405 ms)
2025-02-18 13:18:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926584405 ms)
2025-02-18 13:18:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:18:25.400180+00:00', 'runtime': '926580.001', 'adc_vb_f': '12.674', 'adc_va_f': '19.199', 'adc_vl_f': '12.662', 'adc_ic_f': '0.65', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '14.414', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.199', 'sweep_vmp': '19.196', 'sweep_pmax': '7.881', 'sweep_voc': '22.095', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926584409 ms)
2025-02-18 13:18:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:17:25.400582+00:00', 'runtime': '926520.001', 'adc_vb_f': '12.674', 'adc_va_f': '19.196', 'adc_vl_f': '12.659', 'adc_ic_f': '0.626', 'adc_il_f': '0.343', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '14.414', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '7.957', 'sweep_vmp': '19.196', 'sweep_pmax': '7.881', 'sweep_voc': '22.095', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926584410 ms)
2025-02-18 13:18:35.689 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926593508 ms)
2025-02-18 13:18:35.689 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926593509 ms)
2025-02-18 13:18:35.690 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:18:25.400324+00:00'} (T+926593510 ms)
2025-02-18 13:19:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926644420 ms)
2025-02-18 13:19:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926644420 ms)
2025-02-18 13:19:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:19:25.400199+00:00', 'runtime': '926640.001', 'adc_vb_f': '12.674', 'adc_va_f': '19.199', 'adc_vl_f': '12.662', 'adc_ic_f': '0.672', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '14.414', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.5', 'sweep_vmp': '19.196', 'sweep_pmax': '7.881', 'sweep_voc': '22.095', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926644425 ms)
2025-02-18 13:19:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:18:25.400180+00:00', 'runtime': '926580.001', 'adc_vb_f': '12.674', 'adc_va_f': '19.199', 'adc_vl_f': '12.662', 'adc_ic_f': '0.65', 'adc_il_f': '0.413', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '14.414', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.199', 'sweep_vmp': '19.196', 'sweep_pmax': '7.881', 'sweep_voc': '22.095', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926644425 ms)
2025-02-18 13:20:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926704404 ms)
2025-02-18 13:20:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926704404 ms)
2025-02-18 13:20:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:20:25.400179+00:00', 'runtime': '926700.001', 'adc_vb_f': '12.695', 'adc_va_f': '19.482', 'adc_vl_f': '12.662', 'adc_ic_f': '0.696', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.833', 'sweep_vmp': '19.498', 'sweep_pmax': '8.712', 'sweep_voc': '22.192', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926704409 ms)
2025-02-18 13:20:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:19:25.400199+00:00', 'runtime': '926640.001', 'adc_vb_f': '12.674', 'adc_va_f': '19.199', 'adc_vl_f': '12.662', 'adc_ic_f': '0.672', 'adc_il_f': '0.447', 't_hs': '4', 't_batt': '5', 't_amb': '5', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.674', 'vb_ref': '14.414', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.5', 'sweep_vmp': '19.196', 'sweep_pmax': '7.881', 'sweep_voc': '22.095', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926704409 ms)
2025-02-18 13:21:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926764403 ms)
2025-02-18 13:21:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926764404 ms)
2025-02-18 13:21:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:21:25.400592+00:00', 'runtime': '926760.001', 'adc_vb_f': '12.711', 'adc_va_f': '19.479', 'adc_vl_f': '12.665', 'adc_ic_f': '0.717', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.12', 'sweep_vmp': '19.498', 'sweep_pmax': '8.712', 'sweep_voc': '22.192', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926764405 ms)
2025-02-18 13:21:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:20:25.400179+00:00', 'runtime': '926700.001', 'adc_vb_f': '12.695', 'adc_va_f': '19.482', 'adc_vl_f': '12.662', 'adc_ic_f': '0.696', 'adc_il_f': '0.423', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.689', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1598.9', 'ahl_t': '1598.9', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '8.833', 'sweep_vmp': '19.498', 'sweep_pmax': '8.712', 'sweep_voc': '22.192', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.6', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926764405 ms)
2025-02-18 13:21:52.110 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+926789929 ms)
2025-02-18 13:21:52.110 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+926789929 ms)
2025-02-18 13:21:52.110 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41052)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+926789930 ms)
2025-02-18 13:22:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926824418 ms)
2025-02-18 13:22:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926824419 ms)
2025-02-18 13:22:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:22:25.402771+00:00', 'runtime': '926820.003', 'adc_vb_f': '12.711', 'adc_va_f': '19.482', 'adc_vl_f': '12.692', 'adc_ic_f': '0.739', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.391', 'sweep_vmp': '19.498', 'sweep_pmax': '8.712', 'sweep_voc': '22.192', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926824420 ms)
2025-02-18 13:22:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:21:25.400592+00:00', 'runtime': '926760.001', 'adc_vb_f': '12.711', 'adc_va_f': '19.479', 'adc_vl_f': '12.665', 'adc_ic_f': '0.717', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.704', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.12', 'sweep_vmp': '19.498', 'sweep_pmax': '8.712', 'sweep_voc': '22.192', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926824420 ms)
2025-02-18 13:22:35.903 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926833722 ms)
2025-02-18 13:22:35.903 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926833722 ms)
2025-02-18 13:22:35.904 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:22:25.400149+00:00'} (T+926833723 ms)
2025-02-18 13:23:04.279 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+926862099 ms)
2025-02-18 13:23:04.279 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+926862099 ms)
2025-02-18 13:23:04.280 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41132)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+926862099 ms)
2025-02-18 13:23:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926884419 ms)
2025-02-18 13:23:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926884419 ms)
2025-02-18 13:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:23:25.402800+00:00', 'runtime': '926880.003', 'adc_vb_f': '12.714', 'adc_va_f': '19.73', 'adc_vl_f': '12.698', 'adc_ic_f': '0.761', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.633', 'sweep_vmp': '19.751', 'sweep_pmax': '9.542', 'sweep_voc': '22.25', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926884420 ms)
2025-02-18 13:23:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:22:25.402771+00:00', 'runtime': '926820.003', 'adc_vb_f': '12.711', 'adc_va_f': '19.482', 'adc_vl_f': '12.692', 'adc_ic_f': '0.739', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.391', 'sweep_vmp': '19.498', 'sweep_pmax': '8.712', 'sweep_voc': '22.192', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926884421 ms)
2025-02-18 13:23:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926893426 ms)
2025-02-18 13:23:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926893426 ms)
2025-02-18 13:23:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:23:25.400216+00:00'} (T+926893427 ms)
2025-02-18 13:23:42.370 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+926900189 ms)
2025-02-18 13:23:42.370 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+926900190 ms)
2025-02-18 13:23:42.371 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41174)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+926900190 ms)
2025-02-18 13:23:57.478 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+926915298 ms)
2025-02-18 13:23:57.479 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+926915298 ms)
2025-02-18 13:23:57.479 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41192)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+926915298 ms)
2025-02-18 13:24:13.497 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+926931317 ms)
2025-02-18 13:24:13.498 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+926931317 ms)
2025-02-18 13:24:13.499 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+926931318 ms)
2025-02-18 13:24:25.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+926943414 ms)
2025-02-18 13:24:25.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+926943414 ms)
2025-02-18 13:24:25.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:24:25.403282+00:00', 'runtime': '926940.004', 'adc_vb_f': '12.741', 'adc_va_f': '19.733', 'adc_vl_f': '12.698', 'adc_ic_f': '0.78', 'adc_il_f': '0.355', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.726', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.92', 'sweep_vmp': '19.751', 'sweep_pmax': '9.542', 'sweep_voc': '22.25', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926943416 ms)
2025-02-18 13:24:25.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:23:25.402800+00:00', 'runtime': '926880.003', 'adc_vb_f': '12.714', 'adc_va_f': '19.73', 'adc_vl_f': '12.698', 'adc_ic_f': '0.761', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.711', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.633', 'sweep_vmp': '19.751', 'sweep_pmax': '9.542', 'sweep_voc': '22.25', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+926943417 ms)
2025-02-18 13:24:35.799 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926953618 ms)
2025-02-18 13:24:35.799 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+926953618 ms)
2025-02-18 13:24:35.800 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:24:25.400204+00:00'} (T+926953619 ms)
2025-02-18 13:25:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927004413 ms)
2025-02-18 13:25:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927004413 ms)
2025-02-18 13:25:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:25:25.402588+00:00', 'runtime': '927000.003', 'adc_vb_f': '12.75', 'adc_va_f': '19.733', 'adc_vl_f': '12.698', 'adc_ic_f': '0.8', 'adc_il_f': '0.391', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.744', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.161', 'sweep_vmp': '19.751', 'sweep_pmax': '9.542', 'sweep_voc': '22.25', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927004414 ms)
2025-02-18 13:25:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:24:25.403282+00:00', 'runtime': '926940.004', 'adc_vb_f': '12.741', 'adc_va_f': '19.733', 'adc_vl_f': '12.698', 'adc_ic_f': '0.78', 'adc_il_f': '0.355', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.726', 'vb_ref': '14.384', 'ahc_r': '2660.6', 'ahc_t': '2660.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '9.92', 'sweep_vmp': '19.751', 'sweep_pmax': '9.542', 'sweep_voc': '22.25', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.2', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927004414 ms)
2025-02-18 13:26:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927064416 ms)
2025-02-18 13:26:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927064417 ms)
2025-02-18 13:26:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:26:25.400803+00:00', 'runtime': '927060.001', 'adc_vb_f': '12.75', 'adc_va_f': '19.482', 'adc_vl_f': '12.726', 'adc_ic_f': '0.819', 'adc_il_f': '0.329', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.433', 'sweep_vmp': '19.485', 'sweep_pmax': '10.388', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927064418 ms)
2025-02-18 13:26:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:25:25.402588+00:00', 'runtime': '927000.003', 'adc_vb_f': '12.75', 'adc_va_f': '19.733', 'adc_vl_f': '12.698', 'adc_ic_f': '0.8', 'adc_il_f': '0.391', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.744', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.161', 'sweep_vmp': '19.751', 'sweep_pmax': '9.542', 'sweep_voc': '22.25', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927064418 ms)
2025-02-18 13:27:25.575 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927123394 ms)
2025-02-18 13:27:25.575 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927123395 ms)
2025-02-18 13:27:25.577 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:27:25.402744+00:00', 'runtime': '927120.003', 'adc_vb_f': '12.75', 'adc_va_f': '19.482', 'adc_vl_f': '12.738', 'adc_ic_f': '0.848', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.841', 'sweep_vmp': '19.485', 'sweep_pmax': '10.388', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927123397 ms)
2025-02-18 13:27:25.578 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:26:25.400803+00:00', 'runtime': '927060.001', 'adc_vb_f': '12.75', 'adc_va_f': '19.482', 'adc_vl_f': '12.726', 'adc_ic_f': '0.819', 'adc_il_f': '0.329', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.747', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.433', 'sweep_vmp': '19.485', 'sweep_pmax': '10.388', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927123397 ms)
2025-02-18 13:27:35.592 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927133411 ms)
2025-02-18 13:27:35.592 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927133412 ms)
2025-02-18 13:27:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:27:25.400137+00:00'} (T+927133413 ms)
2025-02-18 13:28:18.867 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+927176686 ms)
2025-02-18 13:28:18.867 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+927176687 ms)
2025-02-18 13:28:18.868 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41478)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+927176687 ms)
2025-02-18 13:28:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927184411 ms)
2025-02-18 13:28:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927184411 ms)
2025-02-18 13:28:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:28:25.402834+00:00', 'runtime': '927180.003', 'adc_vb_f': '12.766', 'adc_va_f': '19.482', 'adc_vl_f': '12.738', 'adc_ic_f': '0.884', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.756', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '11.279', 'sweep_vmp': '19.485', 'sweep_pmax': '10.388', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '12.772', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927184413 ms)
2025-02-18 13:28:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:27:25.402744+00:00', 'runtime': '927120.003', 'adc_vb_f': '12.75', 'adc_va_f': '19.482', 'adc_vl_f': '12.738', 'adc_ic_f': '0.848', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.75', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '10.841', 'sweep_vmp': '19.485', 'sweep_pmax': '10.388', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '12.75', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '2', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927184413 ms)
2025-02-18 13:28:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927193418 ms)
2025-02-18 13:28:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927193418 ms)
2025-02-18 13:28:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:28:25.400231+00:00'} (T+927193419 ms)
2025-02-18 13:29:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927244414 ms)
2025-02-18 13:29:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927244414 ms)
2025-02-18 13:29:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:29:25.400422+00:00', 'runtime': '927240.001', 'adc_vb_f': '12.787', 'adc_va_f': '19.583', 'adc_vl_f': '12.744', 'adc_ic_f': '0.93', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.778', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '11.883', 'sweep_vmp': '19.586', 'sweep_pmax': '11.52', 'sweep_voc': '22.412', 'vb_min_daily': '12.482', 'vb_max_daily': '12.787', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927244415 ms)
2025-02-18 13:29:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:28:25.402834+00:00', 'runtime': '927180.003', 'adc_vb_f': '12.766', 'adc_va_f': '19.482', 'adc_vl_f': '12.738', 'adc_ic_f': '0.884', 'adc_il_f': '0.399', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.756', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '11.279', 'sweep_vmp': '19.485', 'sweep_pmax': '10.388', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '12.772', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927244415 ms)
2025-02-18 13:29:35.438 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927253257 ms)
2025-02-18 13:29:35.439 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927253258 ms)
2025-02-18 13:29:35.440 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:29:25.406320+00:00'} (T+927253259 ms)
2025-02-18 13:30:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927304417 ms)
2025-02-18 13:30:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927304417 ms)
2025-02-18 13:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:30:25.400187+00:00', 'runtime': '927300.001', 'adc_vb_f': '12.787', 'adc_va_f': '19.583', 'adc_vl_f': '12.775', 'adc_ic_f': '0.978', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.471', 'sweep_vmp': '19.586', 'sweep_pmax': '11.52', 'sweep_voc': '22.412', 'vb_min_daily': '12.482', 'vb_max_daily': '12.793', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927304419 ms)
2025-02-18 13:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:29:25.400422+00:00', 'runtime': '927240.001', 'adc_vb_f': '12.787', 'adc_va_f': '19.583', 'adc_vl_f': '12.744', 'adc_ic_f': '0.93', 'adc_il_f': '0.432', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.778', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '11.883', 'sweep_vmp': '19.586', 'sweep_pmax': '11.52', 'sweep_voc': '22.412', 'vb_min_daily': '12.482', 'vb_max_daily': '12.787', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927304420 ms)
2025-02-18 13:30:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927313436 ms)
2025-02-18 13:30:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927313437 ms)
2025-02-18 13:30:35.619 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:30:25.413127+00:00'} (T+927313438 ms)
2025-02-18 13:31:24.194 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+927362014 ms)
2025-02-18 13:31:24.195 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+927362014 ms)
2025-02-18 13:31:24.195 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41676)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+927362014 ms)
2025-02-18 13:31:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927364404 ms)
2025-02-18 13:31:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927364405 ms)
2025-02-18 13:31:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:31:25.400201+00:00', 'runtime': '927360.001', 'adc_vb_f': '12.814', 'adc_va_f': '19.583', 'adc_vl_f': '12.775', 'adc_ic_f': '1.022', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.805', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.06', 'sweep_vmp': '19.586', 'sweep_pmax': '11.52', 'sweep_voc': '22.412', 'vb_min_daily': '12.482', 'vb_max_daily': '12.82', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927364408 ms)
2025-02-18 13:31:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:30:25.400187+00:00', 'runtime': '927300.001', 'adc_vb_f': '12.787', 'adc_va_f': '19.583', 'adc_vl_f': '12.775', 'adc_ic_f': '0.978', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.787', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '12.471', 'sweep_vmp': '19.586', 'sweep_pmax': '11.52', 'sweep_voc': '22.412', 'vb_min_daily': '12.482', 'vb_max_daily': '12.793', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927364408 ms)
2025-02-18 13:31:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927373432 ms)
2025-02-18 13:31:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927373433 ms)
2025-02-18 13:31:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:31:25.400203+00:00'} (T+927373434 ms)
2025-02-18 13:32:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927424405 ms)
2025-02-18 13:32:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927424406 ms)
2025-02-18 13:32:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:32:25.400200+00:00', 'runtime': '927420.001', 'adc_vb_f': '12.823', 'adc_va_f': '19.778', 'adc_vl_f': '12.778', 'adc_ic_f': '1.063', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.82', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.543', 'sweep_vmp': '19.775', 'sweep_pmax': '13.377', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.823', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927424407 ms)
2025-02-18 13:32:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:31:25.400201+00:00', 'runtime': '927360.001', 'adc_vb_f': '12.814', 'adc_va_f': '19.583', 'adc_vl_f': '12.775', 'adc_ic_f': '1.022', 'adc_il_f': '0.428', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.805', 'vb_ref': '14.384', 'ahc_r': '2660.7', 'ahc_t': '2660.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.06', 'sweep_vmp': '19.586', 'sweep_pmax': '11.52', 'sweep_voc': '22.412', 'vb_min_daily': '12.482', 'vb_max_daily': '12.82', 'ahc_daily': '0.3', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927424407 ms)
2025-02-18 13:32:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927433428 ms)
2025-02-18 13:32:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927433428 ms)
2025-02-18 13:32:35.617 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:32:25.400201+00:00'} (T+927433436 ms)
2025-02-18 13:33:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927484408 ms)
2025-02-18 13:33:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927484409 ms)
2025-02-18 13:33:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:33:25.400238+00:00', 'runtime': '927480.001', 'adc_vb_f': '12.839', 'adc_va_f': '19.778', 'adc_vl_f': '12.814', 'adc_ic_f': '1.094', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.827', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.027', 'sweep_vmp': '19.775', 'sweep_pmax': '13.377', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.848', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927484410 ms)
2025-02-18 13:33:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:32:25.400200+00:00', 'runtime': '927420.001', 'adc_vb_f': '12.823', 'adc_va_f': '19.778', 'adc_vl_f': '12.778', 'adc_ic_f': '1.063', 'adc_il_f': '0.445', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.82', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '13.543', 'sweep_vmp': '19.775', 'sweep_pmax': '13.377', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.823', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927484410 ms)
2025-02-18 13:33:35.588 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927493407 ms)
2025-02-18 13:33:35.588 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927493408 ms)
2025-02-18 13:33:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:33:25.400237+00:00'} (T+927493415 ms)
2025-02-18 13:34:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927544411 ms)
2025-02-18 13:34:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927544412 ms)
2025-02-18 13:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:34:25.400199+00:00', 'runtime': '927540.001', 'adc_vb_f': '12.863', 'adc_va_f': '19.781', 'adc_vl_f': '12.842', 'adc_ic_f': '1.121', 'adc_il_f': '0.358', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.854', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.374', 'sweep_vmp': '19.775', 'sweep_pmax': '13.377', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.863', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927544414 ms)
2025-02-18 13:34:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:33:25.400238+00:00', 'runtime': '927480.001', 'adc_vb_f': '12.839', 'adc_va_f': '19.778', 'adc_vl_f': '12.814', 'adc_ic_f': '1.094', 'adc_il_f': '0.401', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.827', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.027', 'sweep_vmp': '19.775', 'sweep_pmax': '13.377', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.848', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927544415 ms)
2025-02-18 13:34:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927553432 ms)
2025-02-18 13:34:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927553432 ms)
2025-02-18 13:34:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:34:25.400217+00:00'} (T+927553439 ms)
2025-02-18 13:35:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927604414 ms)
2025-02-18 13:35:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927604414 ms)
2025-02-18 13:35:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:35:25.400206+00:00', 'runtime': '927600.001', 'adc_vb_f': '12.863', 'adc_va_f': '19.684', 'adc_vl_f': '12.851', 'adc_ic_f': '1.138', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.6', 'sweep_vmp': '19.711', 'sweep_pmax': '14.631', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.863', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927604416 ms)
2025-02-18 13:35:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:34:25.400199+00:00', 'runtime': '927540.001', 'adc_vb_f': '12.863', 'adc_va_f': '19.781', 'adc_vl_f': '12.842', 'adc_ic_f': '1.121', 'adc_il_f': '0.358', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.854', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.374', 'sweep_vmp': '19.775', 'sweep_pmax': '13.377', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.863', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927604416 ms)
2025-02-18 13:35:35.960 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927613779 ms)
2025-02-18 13:35:35.960 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927613780 ms)
2025-02-18 13:35:35.964 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:35:25.400209+00:00'} (T+927613783 ms)
2025-02-18 13:35:48.575 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+927626395 ms)
2025-02-18 13:35:48.576 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+927626395 ms)
2025-02-18 13:35:48.576 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 41968)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+927626395 ms)
2025-02-18 13:36:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927664418 ms)
2025-02-18 13:36:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927664419 ms)
2025-02-18 13:36:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:36:25.400198+00:00', 'runtime': '927660.001', 'adc_vb_f': '12.888', 'adc_va_f': '19.684', 'adc_vl_f': '12.851', 'adc_ic_f': '1.15', 'adc_il_f': '0.374', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.872', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.782', 'sweep_vmp': '19.711', 'sweep_pmax': '14.631', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.891', 'ahc_daily': '0.4', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927664422 ms)
2025-02-18 13:36:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:35:25.400206+00:00', 'runtime': '927600.001', 'adc_vb_f': '12.863', 'adc_va_f': '19.684', 'adc_vl_f': '12.851', 'adc_ic_f': '1.138', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.863', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.0', 'ahl_t': '1599.0', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.6', 'sweep_vmp': '19.711', 'sweep_pmax': '14.631', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.863', 'ahc_daily': '0.4', 'ahl_daily': '5.7', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927664422 ms)
2025-02-18 13:37:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927724407 ms)
2025-02-18 13:37:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927724407 ms)
2025-02-18 13:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:37:25.400588+00:00', 'runtime': '927720.001', 'adc_vb_f': '12.9', 'adc_va_f': '19.681', 'adc_vl_f': '12.851', 'adc_ic_f': '1.157', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.894', 'vb_ref': '14.384', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.887', 'sweep_vmp': '19.711', 'sweep_pmax': '14.631', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.9', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927724411 ms)
2025-02-18 13:37:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:36:25.400198+00:00', 'runtime': '927660.001', 'adc_vb_f': '12.888', 'adc_va_f': '19.684', 'adc_vl_f': '12.851', 'adc_ic_f': '1.15', 'adc_il_f': '0.374', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.872', 'vb_ref': '14.384', 'ahc_r': '2660.8', 'ahc_t': '2660.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.782', 'sweep_vmp': '19.711', 'sweep_pmax': '14.631', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.891', 'ahc_daily': '0.4', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927724412 ms)
2025-02-18 13:37:57.818 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+927755637 ms)
2025-02-18 13:37:57.818 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+927755637 ms)
2025-02-18 13:37:57.818 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42108)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+927755638 ms)
2025-02-18 13:38:12.929 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+927770749 ms)
2025-02-18 13:38:12.930 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+927770749 ms)
2025-02-18 13:38:12.930 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42124)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+927770749 ms)
2025-02-18 13:38:26.608 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927784428 ms)
2025-02-18 13:38:26.609 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927784428 ms)
2025-02-18 13:38:26.612 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:38:25.400507+00:00', 'runtime': '927780.001', 'adc_vb_f': '12.9', 'adc_va_f': '19.83', 'adc_vl_f': '12.888', 'adc_ic_f': '1.162', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.384', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.948', 'sweep_vmp': '19.849', 'sweep_pmax': '15.068', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.9', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927784431 ms)
2025-02-18 13:38:26.613 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:37:25.400588+00:00', 'runtime': '927720.001', 'adc_vb_f': '12.9', 'adc_va_f': '19.681', 'adc_vl_f': '12.851', 'adc_ic_f': '1.157', 'adc_il_f': '0.396', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.894', 'vb_ref': '14.384', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.887', 'sweep_vmp': '19.711', 'sweep_pmax': '14.631', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.9', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927784432 ms)
2025-02-18 13:38:30.431 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
 (T+927788250 ms)
2025-02-18 13:39:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927844415 ms)
2025-02-18 13:39:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927844415 ms)
2025-02-18 13:39:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:39:25.400743+00:00', 'runtime': '927840.001', 'adc_vb_f': '12.903', 'adc_va_f': '19.83', 'adc_vl_f': '12.891', 'adc_ic_f': '1.167', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.023', 'sweep_vmp': '19.849', 'sweep_pmax': '15.068', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.906', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927844416 ms)
2025-02-18 13:39:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:38:25.400507+00:00', 'runtime': '927780.001', 'adc_vb_f': '12.9', 'adc_va_f': '19.83', 'adc_vl_f': '12.888', 'adc_ic_f': '1.162', 'adc_il_f': '0.418', 't_hs': '5', 't_batt': '6', 't_amb': '6', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.384', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '14.948', 'sweep_vmp': '19.849', 'sweep_pmax': '15.068', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.9', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927844417 ms)
2025-02-18 13:40:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927904419 ms)
2025-02-18 13:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927904420 ms)
2025-02-18 13:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:40:25.400200+00:00', 'runtime': '927900.001', 'adc_vb_f': '12.924', 'adc_va_f': '19.827', 'adc_vl_f': '12.891', 'adc_ic_f': '1.191', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.915', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.401', 'sweep_vmp': '19.849', 'sweep_pmax': '15.068', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.936', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927904421 ms)
2025-02-18 13:40:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:39:25.400743+00:00', 'runtime': '927840.001', 'adc_vb_f': '12.903', 'adc_va_f': '19.83', 'adc_vl_f': '12.891', 'adc_ic_f': '1.167', 'adc_il_f': '0.394', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.9', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.023', 'sweep_vmp': '19.849', 'sweep_pmax': '15.068', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.906', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927904421 ms)
2025-02-18 13:40:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927913428 ms)
2025-02-18 13:40:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927913429 ms)
2025-02-18 13:40:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:40:25.413052+00:00'} (T+927913430 ms)
2025-02-18 13:41:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+927964407 ms)
2025-02-18 13:41:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+927964408 ms)
2025-02-18 13:41:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:41:25.400180+00:00', 'runtime': '927960.001', 'adc_vb_f': '12.939', 'adc_va_f': '19.876', 'adc_vl_f': '12.909', 'adc_ic_f': '1.215', 'adc_il_f': '0.413', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.933', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.718', 'sweep_vmp': '19.901', 'sweep_pmax': '15.582', 'sweep_voc': '22.54', 'vb_min_daily': '12.482', 'vb_max_daily': '12.939', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927964410 ms)
2025-02-18 13:41:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:40:25.400200+00:00', 'runtime': '927900.001', 'adc_vb_f': '12.924', 'adc_va_f': '19.827', 'adc_vl_f': '12.891', 'adc_ic_f': '1.191', 'adc_il_f': '0.42', 't_hs': '5', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.915', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.401', 'sweep_vmp': '19.849', 'sweep_pmax': '15.068', 'sweep_voc': '22.522', 'vb_min_daily': '12.482', 'vb_max_daily': '12.936', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+927964410 ms)
2025-02-18 13:41:35.642 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927973461 ms)
2025-02-18 13:41:35.643 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+927973462 ms)
2025-02-18 13:41:35.644 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:41:25.405138+00:00'} (T+927973463 ms)
2025-02-18 13:42:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928024409 ms)
2025-02-18 13:42:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928024409 ms)
2025-02-18 13:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:42:25.401177+00:00', 'runtime': '928020.002', 'adc_vb_f': '12.939', 'adc_va_f': '19.876', 'adc_vl_f': '12.924', 'adc_ic_f': '1.237', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.939', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.004', 'sweep_vmp': '19.901', 'sweep_pmax': '15.582', 'sweep_voc': '22.54', 'vb_min_daily': '12.482', 'vb_max_daily': '12.943', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928024411 ms)
2025-02-18 13:42:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:41:25.400180+00:00', 'runtime': '927960.001', 'adc_vb_f': '12.939', 'adc_va_f': '19.876', 'adc_vl_f': '12.909', 'adc_ic_f': '1.215', 'adc_il_f': '0.413', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.933', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '15.718', 'sweep_vmp': '19.901', 'sweep_pmax': '15.582', 'sweep_voc': '22.54', 'vb_min_daily': '12.482', 'vb_max_daily': '12.939', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928024411 ms)
2025-02-18 13:42:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928033429 ms)
2025-02-18 13:42:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928033430 ms)
2025-02-18 13:42:35.612 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:42:25.400176+00:00'} (T+928033431 ms)
2025-02-18 13:43:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928084410 ms)
2025-02-18 13:43:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928084411 ms)
2025-02-18 13:43:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:43:25.402784+00:00', 'runtime': '928080.003', 'adc_vb_f': '12.958', 'adc_va_f': '19.876', 'adc_vl_f': '12.927', 'adc_ic_f': '1.254', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.952', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.246', 'sweep_vmp': '19.901', 'sweep_pmax': '15.582', 'sweep_voc': '22.54', 'vb_min_daily': '12.482', 'vb_max_daily': '12.967', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928084412 ms)
2025-02-18 13:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:42:25.401177+00:00', 'runtime': '928020.002', 'adc_vb_f': '12.939', 'adc_va_f': '19.876', 'adc_vl_f': '12.924', 'adc_ic_f': '1.237', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.939', 'vb_ref': '14.355', 'ahc_r': '2660.9', 'ahc_t': '2660.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.004', 'sweep_vmp': '19.901', 'sweep_pmax': '15.582', 'sweep_voc': '22.54', 'vb_min_daily': '12.482', 'vb_max_daily': '12.943', 'ahc_daily': '0.5', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928084413 ms)
2025-02-18 13:43:35.721 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928093540 ms)
2025-02-18 13:43:35.721 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928093540 ms)
2025-02-18 13:43:35.722 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:43:25.400220+00:00'} (T+928093541 ms)
2025-02-18 13:44:13.436 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+928131255 ms)
2025-02-18 13:44:13.436 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+928131256 ms)
2025-02-18 13:44:13.437 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42506)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+928131256 ms)
2025-02-18 13:44:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928144413 ms)
2025-02-18 13:44:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928144413 ms)
2025-02-18 13:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:44:25.402675+00:00', 'runtime': '928140.003', 'adc_vb_f': '12.976', 'adc_va_f': '19.778', 'adc_vl_f': '12.936', 'adc_ic_f': '1.28', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.967', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.608', 'sweep_vmp': '19.785', 'sweep_pmax': '16.488', 'sweep_voc': '22.607', 'vb_min_daily': '12.482', 'vb_max_daily': '12.976', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928144415 ms)
2025-02-18 13:44:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:43:25.402784+00:00', 'runtime': '928080.003', 'adc_vb_f': '12.958', 'adc_va_f': '19.876', 'adc_vl_f': '12.927', 'adc_ic_f': '1.254', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.952', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.246', 'sweep_vmp': '19.901', 'sweep_pmax': '15.582', 'sweep_voc': '22.54', 'vb_min_daily': '12.482', 'vb_max_daily': '12.967', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928144416 ms)
2025-02-18 13:44:35.483 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928153302 ms)
2025-02-18 13:44:35.484 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928153303 ms)
2025-02-18 13:44:35.485 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:44:25.400063+00:00'} (T+928153304 ms)
2025-02-18 13:45:18.567 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+928196386 ms)
2025-02-18 13:45:18.567 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+928196386 ms)
2025-02-18 13:45:18.567 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42576)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+928196387 ms)
2025-02-18 13:45:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928204417 ms)
2025-02-18 13:45:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928204418 ms)
2025-02-18 13:45:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:45:25.400164+00:00', 'runtime': '928200.001', 'adc_vb_f': '12.976', 'adc_va_f': '19.778', 'adc_vl_f': '12.961', 'adc_ic_f': '1.283', 'adc_il_f': '0.35', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.976', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.608', 'sweep_vmp': '19.785', 'sweep_pmax': '16.488', 'sweep_voc': '22.607', 'vb_min_daily': '12.482', 'vb_max_daily': '12.976', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928204419 ms)
2025-02-18 13:45:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:44:25.402675+00:00', 'runtime': '928140.003', 'adc_vb_f': '12.976', 'adc_va_f': '19.778', 'adc_vl_f': '12.936', 'adc_ic_f': '1.28', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.967', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.608', 'sweep_vmp': '19.785', 'sweep_pmax': '16.488', 'sweep_voc': '22.607', 'vb_min_daily': '12.482', 'vb_max_daily': '12.976', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928204420 ms)
2025-02-18 13:45:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928213426 ms)
2025-02-18 13:45:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928213426 ms)
2025-02-18 13:45:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:45:25.413125+00:00'} (T+928213427 ms)
2025-02-18 13:46:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928264420 ms)
2025-02-18 13:46:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928264421 ms)
2025-02-18 13:46:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:46:25.400172+00:00', 'runtime': '928260.001', 'adc_vb_f': '12.976', 'adc_va_f': '19.775', 'adc_vl_f': '12.967', 'adc_ic_f': '1.307', 'adc_il_f': '0.345', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.976', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.895', 'sweep_vmp': '19.785', 'sweep_pmax': '16.488', 'sweep_voc': '22.607', 'vb_min_daily': '12.482', 'vb_max_daily': '12.976', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928264422 ms)
2025-02-18 13:46:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:45:25.400164+00:00', 'runtime': '928200.001', 'adc_vb_f': '12.976', 'adc_va_f': '19.778', 'adc_vl_f': '12.961', 'adc_ic_f': '1.283', 'adc_il_f': '0.35', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.976', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.608', 'sweep_vmp': '19.785', 'sweep_pmax': '16.488', 'sweep_voc': '22.607', 'vb_min_daily': '12.482', 'vb_max_daily': '12.976', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928264423 ms)
2025-02-18 13:46:35.634 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928273453 ms)
2025-02-18 13:46:35.635 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928273454 ms)
2025-02-18 13:46:35.636 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:46:25.413121+00:00'} (T+928273455 ms)
2025-02-18 13:47:25.650 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928323469 ms)
2025-02-18 13:47:25.650 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928323470 ms)
2025-02-18 13:47:25.652 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:47:25.402728+00:00', 'runtime': '928320.003', 'adc_vb_f': '13.013', 'adc_va_f': '19.968', 'adc_vl_f': '12.967', 'adc_ic_f': '1.36', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.994', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.68', 'sweep_vmp': '19.992', 'sweep_pmax': '17.273', 'sweep_voc': '22.62', 'vb_min_daily': '12.482', 'vb_max_daily': '13.016', 'ahc_daily': '0.7', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928323472 ms)
2025-02-18 13:47:25.653 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:46:25.400172+00:00', 'runtime': '928260.001', 'adc_vb_f': '12.976', 'adc_va_f': '19.775', 'adc_vl_f': '12.967', 'adc_ic_f': '1.307', 'adc_il_f': '0.345', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.976', 'vb_ref': '14.355', 'ahc_r': '2661.0', 'ahc_t': '2661.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '16.895', 'sweep_vmp': '19.785', 'sweep_pmax': '16.488', 'sweep_voc': '22.607', 'vb_min_daily': '12.482', 'vb_max_daily': '12.976', 'ahc_daily': '0.6', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928323472 ms)
2025-02-18 13:48:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928384409 ms)
2025-02-18 13:48:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928384410 ms)
2025-02-18 13:48:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:48:25.400672+00:00', 'runtime': '928380.001', 'adc_vb_f': '13.016', 'adc_va_f': '19.968', 'adc_vl_f': '12.997', 'adc_ic_f': '1.379', 'adc_il_f': '0.37', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.013', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.952', 'sweep_vmp': '19.992', 'sweep_pmax': '17.273', 'sweep_voc': '22.62', 'vb_min_daily': '12.482', 'vb_max_daily': '13.016', 'ahc_daily': '0.7', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928384411 ms)
2025-02-18 13:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:47:25.402728+00:00', 'runtime': '928320.003', 'adc_vb_f': '13.013', 'adc_va_f': '19.968', 'adc_vl_f': '12.967', 'adc_ic_f': '1.36', 'adc_il_f': '0.384', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '12.994', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.68', 'sweep_vmp': '19.992', 'sweep_pmax': '17.273', 'sweep_voc': '22.62', 'vb_min_daily': '12.482', 'vb_max_daily': '13.016', 'ahc_daily': '0.7', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928384411 ms)
2025-02-18 13:49:08.920 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+928426739 ms)
2025-02-18 13:49:08.920 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+928426740 ms)
2025-02-18 13:49:08.921 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 42832)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+928426740 ms)
2025-02-18 13:49:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928444413 ms)
2025-02-18 13:49:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928444414 ms)
2025-02-18 13:49:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:49:25.400194+00:00', 'runtime': '928440.001', 'adc_vb_f': '13.016', 'adc_va_f': '19.971', 'adc_vl_f': '13.004', 'adc_ic_f': '1.394', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.019', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.133', 'sweep_vmp': '19.992', 'sweep_pmax': '17.273', 'sweep_voc': '22.62', 'vb_min_daily': '12.482', 'vb_max_daily': '13.031', 'ahc_daily': '0.7', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928444414 ms)
2025-02-18 13:49:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:48:25.400672+00:00', 'runtime': '928380.001', 'adc_vb_f': '13.016', 'adc_va_f': '19.968', 'adc_vl_f': '12.997', 'adc_ic_f': '1.379', 'adc_il_f': '0.37', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.013', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '17.952', 'sweep_vmp': '19.992', 'sweep_pmax': '17.273', 'sweep_voc': '22.62', 'vb_min_daily': '12.482', 'vb_max_daily': '13.016', 'ahc_daily': '0.7', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928444415 ms)
2025-02-18 13:49:35.615 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928453434 ms)
2025-02-18 13:49:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928453435 ms)
2025-02-18 13:49:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:49:25.405063+00:00'} (T+928453437 ms)
2025-02-18 13:50:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928504418 ms)
2025-02-18 13:50:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928504419 ms)
2025-02-18 13:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:50:25.402868+00:00', 'runtime': '928500.003', 'adc_vb_f': '13.052', 'adc_va_f': '20.169', 'adc_vl_f': '13.004', 'adc_ic_f': '1.442', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.04', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.783', 'sweep_vmp': '20.193', 'sweep_pmax': '18.42', 'sweep_voc': '22.638', 'vb_min_daily': '12.482', 'vb_max_daily': '13.052', 'ahc_daily': '0.7', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928504419 ms)
2025-02-18 13:50:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:49:25.400194+00:00', 'runtime': '928440.001', 'adc_vb_f': '13.016', 'adc_va_f': '19.971', 'adc_vl_f': '13.004', 'adc_ic_f': '1.394', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.019', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.1', 'ahl_t': '1599.1', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.133', 'sweep_vmp': '19.992', 'sweep_pmax': '17.273', 'sweep_voc': '22.62', 'vb_min_daily': '12.482', 'vb_max_daily': '13.031', 'ahc_daily': '0.7', 'ahl_daily': '5.8', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928504420 ms)
2025-02-18 13:50:35.610 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928513429 ms)
2025-02-18 13:50:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928513430 ms)
2025-02-18 13:50:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:50:25.400256+00:00'} (T+928513432 ms)
2025-02-18 13:51:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928564403 ms)
2025-02-18 13:51:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928564404 ms)
2025-02-18 13:51:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:51:25.400176+00:00', 'runtime': '928560.001', 'adc_vb_f': '13.052', 'adc_va_f': '20.169', 'adc_vl_f': '13.043', 'adc_ic_f': '1.476', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.052', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.22', 'sweep_vmp': '20.193', 'sweep_pmax': '18.42', 'sweep_voc': '22.638', 'vb_min_daily': '12.482', 'vb_max_daily': '13.052', 'ahc_daily': '0.7', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928564405 ms)
2025-02-18 13:51:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:50:25.402868+00:00', 'runtime': '928500.003', 'adc_vb_f': '13.052', 'adc_va_f': '20.169', 'adc_vl_f': '13.004', 'adc_ic_f': '1.442', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.04', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '18.783', 'sweep_vmp': '20.193', 'sweep_pmax': '18.42', 'sweep_voc': '22.638', 'vb_min_daily': '12.482', 'vb_max_daily': '13.052', 'ahc_daily': '0.7', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928564406 ms)
2025-02-18 13:51:35.591 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928573410 ms)
2025-02-18 13:51:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928573410 ms)
2025-02-18 13:51:35.592 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:51:25.400179+00:00'} (T+928573411 ms)
2025-02-18 13:52:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928624409 ms)
2025-02-18 13:52:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928624410 ms)
2025-02-18 13:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:52:25.400240+00:00', 'runtime': '928620.001', 'adc_vb_f': '13.058', 'adc_va_f': '20.169', 'adc_vl_f': '13.043', 'adc_ic_f': '1.486', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.058', 'vb_ref': '14.355', 'ahc_r': '2661.2', 'ahc_t': '2661.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.387', 'sweep_vmp': '20.193', 'sweep_pmax': '18.42', 'sweep_voc': '22.638', 'vb_min_daily': '12.482', 'vb_max_daily': '13.077', 'ahc_daily': '0.8', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928624411 ms)
2025-02-18 13:52:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:51:25.400176+00:00', 'runtime': '928560.001', 'adc_vb_f': '13.052', 'adc_va_f': '20.169', 'adc_vl_f': '13.043', 'adc_ic_f': '1.476', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.052', 'vb_ref': '14.355', 'ahc_r': '2661.1', 'ahc_t': '2661.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.22', 'sweep_vmp': '20.193', 'sweep_pmax': '18.42', 'sweep_voc': '22.638', 'vb_min_daily': '12.482', 'vb_max_daily': '13.052', 'ahc_daily': '0.7', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928624411 ms)
2025-02-18 13:52:36.136 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928633955 ms)
2025-02-18 13:52:36.136 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928633955 ms)
2025-02-18 13:52:36.137 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:52:25.400264+00:00'} (T+928633956 ms)
2025-02-18 13:53:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928684412 ms)
2025-02-18 13:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928684413 ms)
2025-02-18 13:53:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:53:25.400191+00:00', 'runtime': '928680.001', 'adc_vb_f': '13.092', 'adc_va_f': '20.312', 'adc_vl_f': '13.074', 'adc_ic_f': '1.597', 'adc_il_f': '0.428', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.086', 'vb_ref': '14.355', 'ahc_r': '2661.2', 'ahc_t': '2661.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '20.896', 'sweep_vmp': '20.331', 'sweep_pmax': '20.217', 'sweep_voc': '22.729', 'vb_min_daily': '12.482', 'vb_max_daily': '13.092', 'ahc_daily': '0.8', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928684414 ms)
2025-02-18 13:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:52:25.400240+00:00', 'runtime': '928620.001', 'adc_vb_f': '13.058', 'adc_va_f': '20.169', 'adc_vl_f': '13.043', 'adc_ic_f': '1.486', 'adc_il_f': '0.44', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.058', 'vb_ref': '14.355', 'ahc_r': '2661.2', 'ahc_t': '2661.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '19.387', 'sweep_vmp': '20.193', 'sweep_pmax': '18.42', 'sweep_voc': '22.638', 'vb_min_daily': '12.482', 'vb_max_daily': '13.077', 'ahc_daily': '0.8', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928684414 ms)
2025-02-18 13:53:35.798 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928693617 ms)
2025-02-18 13:53:35.799 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+928693619 ms)
2025-02-18 13:53:35.802 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:53:25.400183+00:00'} (T+928693621 ms)
2025-02-18 13:54:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928744416 ms)
2025-02-18 13:54:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928744417 ms)
2025-02-18 13:54:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:54:25.400015+00:00', 'runtime': '928740.001', 'adc_vb_f': '13.126', 'adc_va_f': '20.312', 'adc_vl_f': '13.113', 'adc_ic_f': '1.674', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.113', 'vb_ref': '14.355', 'ahc_r': '2661.2', 'ahc_t': '2661.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '21.953', 'sweep_vmp': '20.331', 'sweep_pmax': '20.217', 'sweep_voc': '22.729', 'vb_min_daily': '12.482', 'vb_max_daily': '13.129', 'ahc_daily': '0.8', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928744420 ms)
2025-02-18 13:54:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:53:25.400191+00:00', 'runtime': '928680.001', 'adc_vb_f': '13.092', 'adc_va_f': '20.312', 'adc_vl_f': '13.074', 'adc_ic_f': '1.597', 'adc_il_f': '0.428', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.086', 'vb_ref': '14.355', 'ahc_r': '2661.2', 'ahc_t': '2661.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20144', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '20.896', 'sweep_vmp': '20.331', 'sweep_pmax': '20.217', 'sweep_voc': '22.729', 'vb_min_daily': '12.482', 'vb_max_daily': '13.092', 'ahc_daily': '0.8', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928744420 ms)
2025-02-18 13:55:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928804405 ms)
2025-02-18 13:55:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928804405 ms)
2025-02-18 13:55:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:55:25.400174+00:00', 'runtime': '928800.001', 'adc_vb_f': '13.138', 'adc_va_f': '20.312', 'adc_vl_f': '13.12', 'adc_ic_f': '1.766', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.132', 'vb_ref': '14.355', 'ahc_r': '2661.3', 'ahc_t': '2661.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.222', 'sweep_vmp': '20.331', 'sweep_pmax': '20.217', 'sweep_voc': '22.729', 'vb_min_daily': '12.482', 'vb_max_daily': '13.15', 'ahc_daily': '0.9', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928804406 ms)
2025-02-18 13:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:54:25.400015+00:00', 'runtime': '928740.001', 'adc_vb_f': '13.126', 'adc_va_f': '20.312', 'adc_vl_f': '13.113', 'adc_ic_f': '1.674', 'adc_il_f': '0.42', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.113', 'vb_ref': '14.355', 'ahc_r': '2661.2', 'ahc_t': '2661.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '21.953', 'sweep_vmp': '20.331', 'sweep_pmax': '20.217', 'sweep_voc': '22.729', 'vb_min_daily': '12.482', 'vb_max_daily': '13.129', 'ahc_daily': '0.8', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928804407 ms)
2025-02-18 13:56:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928864410 ms)
2025-02-18 13:56:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928864411 ms)
2025-02-18 13:56:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:56:25.400623+00:00', 'runtime': '928860.001', 'adc_vb_f': '13.168', 'adc_va_f': '20.316', 'adc_vl_f': '13.153', 'adc_ic_f': '1.814', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.162', 'vb_ref': '14.355', 'ahc_r': '2661.3', 'ahc_t': '2661.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.916', 'sweep_vmp': '20.334', 'sweep_pmax': '23.463', 'sweep_voc': '22.839', 'vb_min_daily': '12.482', 'vb_max_daily': '13.168', 'ahc_daily': '0.9', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928864412 ms)
2025-02-18 13:56:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:55:25.400174+00:00', 'runtime': '928800.001', 'adc_vb_f': '13.138', 'adc_va_f': '20.312', 'adc_vl_f': '13.12', 'adc_ic_f': '1.766', 'adc_il_f': '0.423', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.132', 'vb_ref': '14.355', 'ahc_r': '2661.3', 'ahc_t': '2661.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.222', 'sweep_vmp': '20.331', 'sweep_pmax': '20.217', 'sweep_voc': '22.729', 'vb_min_daily': '12.482', 'vb_max_daily': '13.15', 'ahc_daily': '0.9', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928864412 ms)
2025-02-18 13:57:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928924413 ms)
2025-02-18 13:57:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928924414 ms)
2025-02-18 13:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:57:25.402783+00:00', 'runtime': '928920.003', 'adc_vb_f': '13.242', 'adc_va_f': '20.316', 'adc_vl_f': '13.202', 'adc_ic_f': '2.143', 'adc_il_f': '0.401', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.205', 'vb_ref': '14.355', 'ahc_r': '2661.3', 'ahc_t': '2661.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '28.385', 'sweep_vmp': '20.334', 'sweep_pmax': '23.463', 'sweep_voc': '22.839', 'vb_min_daily': '12.482', 'vb_max_daily': '13.245', 'ahc_daily': '0.9', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928924415 ms)
2025-02-18 13:57:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:56:25.400623+00:00', 'runtime': '928860.001', 'adc_vb_f': '13.168', 'adc_va_f': '20.316', 'adc_vl_f': '13.153', 'adc_ic_f': '1.814', 'adc_il_f': '0.416', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.162', 'vb_ref': '14.355', 'ahc_r': '2661.3', 'ahc_t': '2661.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '23.916', 'sweep_vmp': '20.334', 'sweep_pmax': '23.463', 'sweep_voc': '22.839', 'vb_min_daily': '12.482', 'vb_max_daily': '13.168', 'ahc_daily': '0.9', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928924415 ms)
2025-02-18 13:58:04.713 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+928962532 ms)
2025-02-18 13:58:04.713 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+928962533 ms)
2025-02-18 13:58:04.714 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43410)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+928962533 ms)
2025-02-18 13:58:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+928984417 ms)
2025-02-18 13:58:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+928984417 ms)
2025-02-18 13:58:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:58:25.400764+00:00', 'runtime': '928980.001', 'adc_vb_f': '13.281', 'adc_va_f': '20.312', 'adc_vl_f': '13.272', 'adc_ic_f': '2.322', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.266', 'vb_ref': '14.355', 'ahc_r': '2661.4', 'ahc_t': '2661.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '30.756', 'sweep_vmp': '20.334', 'sweep_pmax': '23.463', 'sweep_voc': '22.839', 'vb_min_daily': '12.482', 'vb_max_daily': '13.281', 'ahc_daily': '1.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928984418 ms)
2025-02-18 13:58:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:57:25.402783+00:00', 'runtime': '928920.003', 'adc_vb_f': '13.242', 'adc_va_f': '20.316', 'adc_vl_f': '13.202', 'adc_ic_f': '2.143', 'adc_il_f': '0.401', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.205', 'vb_ref': '14.355', 'ahc_r': '2661.3', 'ahc_t': '2661.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '28.385', 'sweep_vmp': '20.334', 'sweep_pmax': '23.463', 'sweep_voc': '22.839', 'vb_min_daily': '12.482', 'vb_max_daily': '13.245', 'ahc_daily': '0.9', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+928984419 ms)
2025-02-18 13:59:25.645 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929043464 ms)
2025-02-18 13:59:25.646 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929043465 ms)
2025-02-18 13:59:25.647 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 13:59:25.402839+00:00', 'runtime': '929040.003', 'adc_vb_f': '13.3', 'adc_va_f': '20.505', 'adc_vl_f': '13.272', 'adc_ic_f': '2.104', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.297', 'vb_ref': '14.355', 'ahc_r': '2661.4', 'ahc_t': '2661.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '27.978', 'sweep_vmp': '20.511', 'sweep_pmax': '29.563', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.315', 'ahc_daily': '1.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929043466 ms)
2025-02-18 13:59:25.647 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:58:25.400764+00:00', 'runtime': '928980.001', 'adc_vb_f': '13.281', 'adc_va_f': '20.312', 'adc_vl_f': '13.272', 'adc_ic_f': '2.322', 'adc_il_f': '0.432', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.266', 'vb_ref': '14.355', 'ahc_r': '2661.4', 'ahc_t': '2661.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '30.756', 'sweep_vmp': '20.334', 'sweep_pmax': '23.463', 'sweep_voc': '22.839', 'vb_min_daily': '12.482', 'vb_max_daily': '13.281', 'ahc_daily': '1.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929043466 ms)
2025-02-18 13:59:35.596 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929053415 ms)
2025-02-18 13:59:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929053416 ms)
2025-02-18 13:59:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 13:59:25.400183+00:00'} (T+929053417 ms)
2025-02-18 13:59:35.888 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929053707 ms)
2025-02-18 13:59:35.888 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929053707 ms)
2025-02-18 13:59:35.889 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43506)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929053708 ms)
2025-02-18 14:00:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929104416 ms)
2025-02-18 14:00:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929104417 ms)
2025-02-18 14:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:00:25.400453+00:00', 'runtime': '929100.001', 'adc_vb_f': '13.354', 'adc_va_f': '20.502', 'adc_vl_f': '13.312', 'adc_ic_f': '2.338', 'adc_il_f': '0.408', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.33', 'vb_ref': '14.325', 'ahc_r': '2661.4', 'ahc_t': '2661.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '31.284', 'sweep_vmp': '20.511', 'sweep_pmax': '29.563', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.358', 'ahc_daily': '1.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929104421 ms)
2025-02-18 14:00:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 13:59:25.402839+00:00', 'runtime': '929040.003', 'adc_vb_f': '13.3', 'adc_va_f': '20.505', 'adc_vl_f': '13.272', 'adc_ic_f': '2.104', 'adc_il_f': '0.435', 't_hs': '6', 't_batt': '7', 't_amb': '7', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.297', 'vb_ref': '14.355', 'ahc_r': '2661.4', 'ahc_t': '2661.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '27.978', 'sweep_vmp': '20.511', 'sweep_pmax': '29.563', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.315', 'ahc_daily': '1.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929104421 ms)
2025-02-18 14:00:35.609 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929113428 ms)
2025-02-18 14:00:35.610 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929113429 ms)
2025-02-18 14:00:35.611 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:00:25.401140+00:00'} (T+929113430 ms)
2025-02-18 14:01:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929164416 ms)
2025-02-18 14:01:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929164417 ms)
2025-02-18 14:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:01:25.400145+00:00', 'runtime': '929160.001', 'adc_vb_f': '13.379', 'adc_va_f': '20.505', 'adc_vl_f': '13.351', 'adc_ic_f': '2.372', 'adc_il_f': '0.43', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.361', 'vb_ref': '14.325', 'ahc_r': '2661.5', 'ahc_t': '2661.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '31.752', 'sweep_vmp': '20.511', 'sweep_pmax': '29.563', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.391', 'ahc_daily': '1.1', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929164421 ms)
2025-02-18 14:01:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:00:25.400453+00:00', 'runtime': '929100.001', 'adc_vb_f': '13.354', 'adc_va_f': '20.502', 'adc_vl_f': '13.312', 'adc_ic_f': '2.338', 'adc_il_f': '0.408', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.33', 'vb_ref': '14.325', 'ahc_r': '2661.4', 'ahc_t': '2661.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '31.284', 'sweep_vmp': '20.511', 'sweep_pmax': '29.563', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.358', 'ahc_daily': '1.0', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929164421 ms)
2025-02-18 14:01:35.621 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929173441 ms)
2025-02-18 14:01:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929173441 ms)
2025-02-18 14:01:35.623 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:01:25.401216+00:00'} (T+929173442 ms)
2025-02-18 14:02:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929224420 ms)
2025-02-18 14:02:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929224420 ms)
2025-02-18 14:02:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:02:25.402863+00:00', 'runtime': '929220.003', 'adc_vb_f': '13.422', 'adc_va_f': '20.599', 'adc_vl_f': '13.385', 'adc_ic_f': '2.438', 'adc_il_f': '0.428', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.397', 'vb_ref': '14.325', 'ahc_r': '2661.5', 'ahc_t': '2661.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '32.704', 'sweep_vmp': '20.609', 'sweep_pmax': '31.994', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.425', 'ahc_daily': '1.1', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929224421 ms)
2025-02-18 14:02:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:01:25.400145+00:00', 'runtime': '929160.001', 'adc_vb_f': '13.379', 'adc_va_f': '20.505', 'adc_vl_f': '13.351', 'adc_ic_f': '2.372', 'adc_il_f': '0.43', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.361', 'vb_ref': '14.325', 'ahc_r': '2661.5', 'ahc_t': '2661.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '31.752', 'sweep_vmp': '20.511', 'sweep_pmax': '29.563', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.391', 'ahc_daily': '1.1', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929224422 ms)
2025-02-18 14:02:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929233437 ms)
2025-02-18 14:02:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929233438 ms)
2025-02-18 14:02:35.620 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:02:25.400249+00:00'} (T+929233439 ms)
2025-02-18 14:03:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929284419 ms)
2025-02-18 14:03:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929284419 ms)
2025-02-18 14:03:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:03:25.400167+00:00', 'runtime': '929280.001', 'adc_vb_f': '13.47', 'adc_va_f': '20.599', 'adc_vl_f': '13.455', 'adc_ic_f': '2.631', 'adc_il_f': '0.408', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.443', 'vb_ref': '14.325', 'ahc_r': '2661.5', 'ahc_t': '2661.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.436', 'sweep_vmp': '20.609', 'sweep_pmax': '31.994', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.47', 'ahc_daily': '1.1', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929284420 ms)
2025-02-18 14:03:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:02:25.402863+00:00', 'runtime': '929220.003', 'adc_vb_f': '13.422', 'adc_va_f': '20.599', 'adc_vl_f': '13.385', 'adc_ic_f': '2.438', 'adc_il_f': '0.428', 't_hs': '6', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.397', 'vb_ref': '14.325', 'ahc_r': '2661.5', 'ahc_t': '2661.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '32.704', 'sweep_vmp': '20.609', 'sweep_pmax': '31.994', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.425', 'ahc_daily': '1.1', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929284421 ms)
2025-02-18 14:03:35.663 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929293482 ms)
2025-02-18 14:03:35.664 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929293483 ms)
2025-02-18 14:03:35.665 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:03:25.405137+00:00'} (T+929293485 ms)
2025-02-18 14:04:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929344406 ms)
2025-02-18 14:04:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929344407 ms)
2025-02-18 14:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:04:25.400124+00:00', 'runtime': '929340.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.596', 'adc_vl_f': '13.501', 'adc_ic_f': '2.669', 'adc_il_f': '0.425', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.498', 'vb_ref': '14.325', 'ahc_r': '2661.6', 'ahc_t': '2661.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.086', 'sweep_vmp': '20.609', 'sweep_pmax': '31.994', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.51', 'ahc_daily': '1.2', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929344411 ms)
2025-02-18 14:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:03:25.400167+00:00', 'runtime': '929280.001', 'adc_vb_f': '13.47', 'adc_va_f': '20.599', 'adc_vl_f': '13.455', 'adc_ic_f': '2.631', 'adc_il_f': '0.408', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.443', 'vb_ref': '14.325', 'ahc_r': '2661.5', 'ahc_t': '2661.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.436', 'sweep_vmp': '20.609', 'sweep_pmax': '31.994', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.47', 'ahc_daily': '1.1', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929344411 ms)
2025-02-18 14:04:35.871 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929353691 ms)
2025-02-18 14:04:35.872 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929353691 ms)
2025-02-18 14:04:35.873 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:04:25.405135+00:00'} (T+929353692 ms)
2025-02-18 14:05:00.365 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929378185 ms)
2025-02-18 14:05:00.366 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929378185 ms)
2025-02-18 14:05:00.366 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 43852)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929378185 ms)
2025-02-18 14:05:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929404410 ms)
2025-02-18 14:05:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929404411 ms)
2025-02-18 14:05:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:05:25.400167+00:00', 'runtime': '929400.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.599', 'adc_vl_f': '13.501', 'adc_ic_f': '2.556', 'adc_il_f': '0.43', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.51', 'vb_ref': '14.325', 'ahc_r': '2661.6', 'ahc_t': '2661.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.485', 'sweep_vmp': '20.633', 'sweep_pmax': '34.863', 'sweep_voc': '22.971', 'vb_min_daily': '12.482', 'vb_max_daily': '13.513', 'ahc_daily': '1.2', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929404415 ms)
2025-02-18 14:05:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:04:25.400124+00:00', 'runtime': '929340.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.596', 'adc_vl_f': '13.501', 'adc_ic_f': '2.669', 'adc_il_f': '0.425', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.498', 'vb_ref': '14.325', 'ahc_r': '2661.6', 'ahc_t': '2661.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.2', 'ahl_t': '1599.2', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.086', 'sweep_vmp': '20.609', 'sweep_pmax': '31.994', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.51', 'ahc_daily': '1.2', 'ahl_daily': '5.9', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929404415 ms)
2025-02-18 14:05:35.709 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929413528 ms)
2025-02-18 14:05:35.709 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929413529 ms)
2025-02-18 14:05:35.710 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:05:25.413126+00:00'} (T+929413529 ms)
2025-02-18 14:06:22.469 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+929460289 ms)
2025-02-18 14:06:22.470 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+929460289 ms)
2025-02-18 14:06:22.470 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929460289 ms)
2025-02-18 14:06:22.471 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+929460290 ms)
2025-02-18 14:06:22.471 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+929460290 ms)
2025-02-18 14:06:22.471 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929460291 ms)
2025-02-18 14:06:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929464413 ms)
2025-02-18 14:06:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929464414 ms)
2025-02-18 14:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:06:25.400156+00:00', 'runtime': '929460.001', 'adc_vb_f': '13.547', 'adc_va_f': '20.599', 'adc_vl_f': '13.538', 'adc_ic_f': '2.662', 'adc_il_f': '0.425', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.541', 'vb_ref': '14.325', 'ahc_r': '2661.7', 'ahc_t': '2661.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.055', 'sweep_vmp': '20.633', 'sweep_pmax': '34.863', 'sweep_voc': '22.971', 'vb_min_daily': '12.482', 'vb_max_daily': '13.55', 'ahc_daily': '1.3', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929464416 ms)
2025-02-18 14:06:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:05:25.400167+00:00', 'runtime': '929400.001', 'adc_vb_f': '13.51', 'adc_va_f': '20.599', 'adc_vl_f': '13.501', 'adc_ic_f': '2.556', 'adc_il_f': '0.43', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.51', 'vb_ref': '14.325', 'ahc_r': '2661.6', 'ahc_t': '2661.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '34.485', 'sweep_vmp': '20.633', 'sweep_pmax': '34.863', 'sweep_voc': '22.971', 'vb_min_daily': '12.482', 'vb_max_daily': '13.513', 'ahc_daily': '1.2', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929464416 ms)
2025-02-18 14:06:35.591 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929473410 ms)
2025-02-18 14:06:35.592 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929473411 ms)
2025-02-18 14:06:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:06:25.400074+00:00'} (T+929473416 ms)
2025-02-18 14:07:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929524417 ms)
2025-02-18 14:07:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929524418 ms)
2025-02-18 14:07:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:07:25.400176+00:00', 'runtime': '929520.001', 'adc_vb_f': '13.583', 'adc_va_f': '20.599', 'adc_vl_f': '13.556', 'adc_ic_f': '2.609', 'adc_il_f': '0.391', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.577', 'vb_ref': '14.325', 'ahc_r': '2661.7', 'ahc_t': '2661.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.467', 'sweep_vmp': '20.633', 'sweep_pmax': '34.863', 'sweep_voc': '22.971', 'vb_min_daily': '12.482', 'vb_max_daily': '13.586', 'ahc_daily': '1.3', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929524419 ms)
2025-02-18 14:07:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:06:25.400156+00:00', 'runtime': '929460.001', 'adc_vb_f': '13.547', 'adc_va_f': '20.599', 'adc_vl_f': '13.538', 'adc_ic_f': '2.662', 'adc_il_f': '0.425', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.541', 'vb_ref': '14.325', 'ahc_r': '2661.7', 'ahc_t': '2661.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.055', 'sweep_vmp': '20.633', 'sweep_pmax': '34.863', 'sweep_voc': '22.971', 'vb_min_daily': '12.482', 'vb_max_daily': '13.55', 'ahc_daily': '1.3', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929524419 ms)
2025-02-18 14:07:35.617 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929533436 ms)
2025-02-18 14:07:35.618 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929533437 ms)
2025-02-18 14:07:35.622 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:07:25.400203+00:00'} (T+929533442 ms)
2025-02-18 14:07:54.653 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929552472 ms)
2025-02-18 14:07:54.653 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929552473 ms)
2025-02-18 14:07:54.654 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44050)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929552473 ms)
2025-02-18 14:08:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929584420 ms)
2025-02-18 14:08:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929584420 ms)
2025-02-18 14:08:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:08:25.400195+00:00', 'runtime': '929580.001', 'adc_vb_f': '13.632', 'adc_va_f': '20.703', 'adc_vl_f': '13.614', 'adc_ic_f': '3.066', 'adc_il_f': '0.362', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.623', 'vb_ref': '14.325', 'ahc_r': '2661.8', 'ahc_t': '2661.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '41.778', 'sweep_vmp': '20.712', 'sweep_pmax': '40.781', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '13.657', 'ahc_daily': '1.4', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929584424 ms)
2025-02-18 14:08:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:07:25.400176+00:00', 'runtime': '929520.001', 'adc_vb_f': '13.583', 'adc_va_f': '20.599', 'adc_vl_f': '13.556', 'adc_ic_f': '2.609', 'adc_il_f': '0.391', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.577', 'vb_ref': '14.325', 'ahc_r': '2661.7', 'ahc_t': '2661.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '35.467', 'sweep_vmp': '20.633', 'sweep_pmax': '34.863', 'sweep_voc': '22.971', 'vb_min_daily': '12.482', 'vb_max_daily': '13.586', 'ahc_daily': '1.3', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929584425 ms)
2025-02-18 14:08:35.588 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929593407 ms)
2025-02-18 14:08:35.589 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929593408 ms)
2025-02-18 14:08:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:08:25.400187+00:00'} (T+929593413 ms)
2025-02-18 14:09:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929644407 ms)
2025-02-18 14:09:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929644408 ms)
2025-02-18 14:09:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:09:25.400153+00:00', 'runtime': '929640.001', 'adc_vb_f': '13.623', 'adc_va_f': '20.706', 'adc_vl_f': '13.614', 'adc_ic_f': '2.819', 'adc_il_f': '0.35', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.623', 'vb_ref': '14.325', 'ahc_r': '2661.8', 'ahc_t': '2661.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '38.365', 'sweep_vmp': '20.712', 'sweep_pmax': '40.781', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '13.657', 'ahc_daily': '1.4', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929644412 ms)
2025-02-18 14:09:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:08:25.400195+00:00', 'runtime': '929580.001', 'adc_vb_f': '13.632', 'adc_va_f': '20.703', 'adc_vl_f': '13.614', 'adc_ic_f': '3.066', 'adc_il_f': '0.362', 't_hs': '7', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.623', 'vb_ref': '14.325', 'ahc_r': '2661.8', 'ahc_t': '2661.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '41.778', 'sweep_vmp': '20.712', 'sweep_pmax': '40.781', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '13.657', 'ahc_daily': '1.4', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929644412 ms)
2025-02-18 14:09:35.577 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929653396 ms)
2025-02-18 14:09:35.577 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929653397 ms)
2025-02-18 14:09:35.579 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:09:25.400209+00:00'} (T+929653398 ms)
2025-02-18 14:10:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929704410 ms)
2025-02-18 14:10:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929704411 ms)
2025-02-18 14:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:10:25.400171+00:00', 'runtime': '929700.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.703', 'adc_vl_f': '13.654', 'adc_ic_f': '2.938', 'adc_il_f': '0.343', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.657', 'vb_ref': '14.296', 'ahc_r': '2661.9', 'ahc_t': '2661.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.132', 'sweep_vmp': '20.712', 'sweep_pmax': '40.781', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '13.663', 'ahc_daily': '1.5', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929704415 ms)
2025-02-18 14:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:09:25.400153+00:00', 'runtime': '929640.001', 'adc_vb_f': '13.623', 'adc_va_f': '20.706', 'adc_vl_f': '13.614', 'adc_ic_f': '2.819', 'adc_il_f': '0.35', 't_hs': '8', 't_batt': '8', 't_amb': '8', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.623', 'vb_ref': '14.325', 'ahc_r': '2661.8', 'ahc_t': '2661.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '38.365', 'sweep_vmp': '20.712', 'sweep_pmax': '40.781', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '13.657', 'ahc_daily': '1.4', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929704415 ms)
2025-02-18 14:11:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929764413 ms)
2025-02-18 14:11:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929764414 ms)
2025-02-18 14:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:11:25.400515+00:00', 'runtime': '929760.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.599', 'adc_vl_f': '13.647', 'adc_ic_f': '2.72', 'adc_il_f': '0.36', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.296', 'ahc_r': '2661.9', 'ahc_t': '2661.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.127', 'sweep_vmp': '20.609', 'sweep_pmax': '39.332', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.687', 'ahc_daily': '1.5', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929764415 ms)
2025-02-18 14:11:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:10:25.400171+00:00', 'runtime': '929700.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.703', 'adc_vl_f': '13.654', 'adc_ic_f': '2.938', 'adc_il_f': '0.343', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.657', 'vb_ref': '14.296', 'ahc_r': '2661.9', 'ahc_t': '2661.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.132', 'sweep_vmp': '20.712', 'sweep_pmax': '40.781', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '13.663', 'ahc_daily': '1.5', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929764415 ms)
2025-02-18 14:11:42.016 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929779835 ms)
2025-02-18 14:11:42.016 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929779835 ms)
2025-02-18 14:11:42.016 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44290)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929779836 ms)
2025-02-18 14:11:45.097 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929782917 ms)
2025-02-18 14:11:45.098 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929782917 ms)
2025-02-18 14:11:45.098 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44294)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929782918 ms)
2025-02-18 14:12:07.178 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929804997 ms)
2025-02-18 14:12:07.178 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929804997 ms)
2025-02-18 14:12:07.179 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44320)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929804998 ms)
2025-02-18 14:12:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929824420 ms)
2025-02-18 14:12:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929824421 ms)
2025-02-18 14:12:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:12:25.400192+00:00', 'runtime': '929820.001', 'adc_vb_f': '13.644', 'adc_va_f': '20.596', 'adc_vl_f': '13.614', 'adc_ic_f': '2.645', 'adc_il_f': '0.353', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.654', 'vb_ref': '14.296', 'ahc_r': '2662.0', 'ahc_t': '2662.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.055', 'sweep_vmp': '20.609', 'sweep_pmax': '39.332', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.687', 'ahc_daily': '1.6', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929824422 ms)
2025-02-18 14:12:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:11:25.400515+00:00', 'runtime': '929760.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.599', 'adc_vl_f': '13.647', 'adc_ic_f': '2.72', 'adc_il_f': '0.36', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.663', 'vb_ref': '14.296', 'ahc_r': '2661.9', 'ahc_t': '2661.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.127', 'sweep_vmp': '20.609', 'sweep_pmax': '39.332', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.687', 'ahc_daily': '1.5', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929824423 ms)
2025-02-18 14:12:35.444 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929833263 ms)
2025-02-18 14:12:35.444 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929833264 ms)
2025-02-18 14:12:35.446 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:12:25.400334+00:00'} (T+929833265 ms)
2025-02-18 14:13:26.604 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929884424 ms)
2025-02-18 14:13:26.605 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929884424 ms)
2025-02-18 14:13:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:13:25.400196+00:00', 'runtime': '929880.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.596', 'adc_vl_f': '13.654', 'adc_ic_f': '2.778', 'adc_il_f': '0.36', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.66', 'vb_ref': '14.296', 'ahc_r': '2662.0', 'ahc_t': '2662.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.913', 'sweep_vmp': '20.609', 'sweep_pmax': '39.332', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.687', 'ahc_daily': '1.6', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929884425 ms)
2025-02-18 14:13:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:12:25.400192+00:00', 'runtime': '929820.001', 'adc_vb_f': '13.644', 'adc_va_f': '20.596', 'adc_vl_f': '13.614', 'adc_ic_f': '2.645', 'adc_il_f': '0.353', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.654', 'vb_ref': '14.296', 'ahc_r': '2662.0', 'ahc_t': '2662.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.055', 'sweep_vmp': '20.609', 'sweep_pmax': '39.332', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.687', 'ahc_daily': '1.6', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929884426 ms)
2025-02-18 14:13:35.656 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929893476 ms)
2025-02-18 14:13:35.657 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929893476 ms)
2025-02-18 14:13:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:13:25.413124+00:00'} (T+929893477 ms)
2025-02-18 14:14:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+929944414 ms)
2025-02-18 14:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+929944414 ms)
2025-02-18 14:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:14:25.402694+00:00', 'runtime': '929940.003', 'adc_vb_f': '13.699', 'adc_va_f': '20.456', 'adc_vl_f': '13.69', 'adc_ic_f': '2.916', 'adc_il_f': '0.35', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.678', 'vb_ref': '14.266', 'ahc_r': '2662.1', 'ahc_t': '2662.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '39.906', 'sweep_vmp': '20.48', 'sweep_pmax': '37.867', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.7', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929944415 ms)
2025-02-18 14:14:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:13:25.400196+00:00', 'runtime': '929880.001', 'adc_vb_f': '13.663', 'adc_va_f': '20.596', 'adc_vl_f': '13.654', 'adc_ic_f': '2.778', 'adc_il_f': '0.36', 't_hs': '8', 't_batt': '9', 't_amb': '9', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.66', 'vb_ref': '14.296', 'ahc_r': '2662.0', 'ahc_t': '2662.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.913', 'sweep_vmp': '20.609', 'sweep_pmax': '39.332', 'sweep_voc': '22.958', 'vb_min_daily': '12.482', 'vb_max_daily': '13.687', 'ahc_daily': '1.6', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+929944415 ms)
2025-02-18 14:14:32.425 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929950244 ms)
2025-02-18 14:14:32.425 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929950244 ms)
2025-02-18 14:14:32.425 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44474)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929950245 ms)
2025-02-18 14:14:35.592 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929953412 ms)
2025-02-18 14:14:35.593 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+929953412 ms)
2025-02-18 14:14:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:14:25.400149+00:00'} (T+929953413 ms)
2025-02-18 14:14:37.493 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+929955312 ms)
2025-02-18 14:14:37.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+929955312 ms)
2025-02-18 14:14:37.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44482)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+929955313 ms)
2025-02-18 14:15:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930004418 ms)
2025-02-18 14:15:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930004418 ms)
2025-02-18 14:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:15:25.400147+00:00', 'runtime': '930000.001', 'adc_vb_f': '13.666', 'adc_va_f': '20.459', 'adc_vl_f': '13.654', 'adc_ic_f': '2.715', 'adc_il_f': '0.355', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.669', 'vb_ref': '14.266', 'ahc_r': '2662.1', 'ahc_t': '2662.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.052', 'sweep_vmp': '20.48', 'sweep_pmax': '37.867', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.7', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930004419 ms)
2025-02-18 14:15:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:14:25.402694+00:00', 'runtime': '929940.003', 'adc_vb_f': '13.699', 'adc_va_f': '20.456', 'adc_vl_f': '13.69', 'adc_ic_f': '2.916', 'adc_il_f': '0.35', 't_hs': '8', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.678', 'vb_ref': '14.266', 'ahc_r': '2662.1', 'ahc_t': '2662.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '39.906', 'sweep_vmp': '20.48', 'sweep_pmax': '37.867', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.7', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930004420 ms)
2025-02-18 14:16:24.672 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930062491 ms)
2025-02-18 14:16:24.672 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930062491 ms)
2025-02-18 14:16:24.672 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44606)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930062492 ms)
2025-02-18 14:16:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930064407 ms)
2025-02-18 14:16:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930064408 ms)
2025-02-18 14:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:16:25.400173+00:00', 'runtime': '930060.001', 'adc_vb_f': '13.681', 'adc_va_f': '20.456', 'adc_vl_f': '13.654', 'adc_ic_f': '2.752', 'adc_il_f': '0.345', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.675', 'vb_ref': '14.266', 'ahc_r': '2662.1', 'ahc_t': '2662.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.641', 'sweep_vmp': '20.48', 'sweep_pmax': '37.867', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.7', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930064411 ms)
2025-02-18 14:16:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:15:25.400147+00:00', 'runtime': '930000.001', 'adc_vb_f': '13.666', 'adc_va_f': '20.459', 'adc_vl_f': '13.654', 'adc_ic_f': '2.715', 'adc_il_f': '0.355', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.669', 'vb_ref': '14.266', 'ahc_r': '2662.1', 'ahc_t': '2662.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.052', 'sweep_vmp': '20.48', 'sweep_pmax': '37.867', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.7', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930064412 ms)
2025-02-18 14:16:43.755 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930081574 ms)
2025-02-18 14:16:43.755 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930081574 ms)
2025-02-18 14:16:43.756 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44626)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930081575 ms)
2025-02-18 14:17:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930124414 ms)
2025-02-18 14:17:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930124415 ms)
2025-02-18 14:17:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:17:25.400183+00:00', 'runtime': '930120.001', 'adc_vb_f': '13.684', 'adc_va_f': '20.361', 'adc_vl_f': '13.654', 'adc_ic_f': '2.706', 'adc_il_f': '0.333', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.678', 'vb_ref': '14.266', 'ahc_r': '2662.2', 'ahc_t': '2662.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.976', 'sweep_vmp': '20.37', 'sweep_pmax': '36.992', 'sweep_voc': '22.925', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.8', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930124417 ms)
2025-02-18 14:17:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:16:25.400173+00:00', 'runtime': '930060.001', 'adc_vb_f': '13.681', 'adc_va_f': '20.456', 'adc_vl_f': '13.654', 'adc_ic_f': '2.752', 'adc_il_f': '0.345', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.675', 'vb_ref': '14.266', 'ahc_r': '2662.1', 'ahc_t': '2662.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '37.641', 'sweep_vmp': '20.48', 'sweep_pmax': '37.867', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.7', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930124417 ms)
2025-02-18 14:17:47.880 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930145700 ms)
2025-02-18 14:17:47.881 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930145700 ms)
2025-02-18 14:17:47.881 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44694)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930145700 ms)
2025-02-18 14:18:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930184419 ms)
2025-02-18 14:18:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930184419 ms)
2025-02-18 14:18:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:18:25.400163+00:00', 'runtime': '930180.001', 'adc_vb_f': '13.739', 'adc_va_f': '20.361', 'adc_vl_f': '13.73', 'adc_ic_f': '2.971', 'adc_il_f': '0.345', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.708', 'vb_ref': '14.266', 'ahc_r': '2662.2', 'ahc_t': '2662.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.811', 'sweep_vmp': '20.37', 'sweep_pmax': '36.992', 'sweep_voc': '22.925', 'vb_min_daily': '12.482', 'vb_max_daily': '13.739', 'ahc_daily': '1.8', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930184420 ms)
2025-02-18 14:18:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:17:25.400183+00:00', 'runtime': '930120.001', 'adc_vb_f': '13.684', 'adc_va_f': '20.361', 'adc_vl_f': '13.654', 'adc_ic_f': '2.706', 'adc_il_f': '0.333', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.678', 'vb_ref': '14.266', 'ahc_r': '2662.2', 'ahc_t': '2662.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '36.976', 'sweep_vmp': '20.37', 'sweep_pmax': '36.992', 'sweep_voc': '22.925', 'vb_min_daily': '12.482', 'vb_max_daily': '13.699', 'ahc_daily': '1.8', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930184421 ms)
2025-02-18 14:19:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930244408 ms)
2025-02-18 14:19:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930244408 ms)
2025-02-18 14:19:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:19:25.400153+00:00', 'runtime': '930240.001', 'adc_vb_f': '13.748', 'adc_va_f': '20.361', 'adc_vl_f': '13.73', 'adc_ic_f': '2.942', 'adc_il_f': '0.348', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.739', 'vb_ref': '14.266', 'ahc_r': '2662.3', 'ahc_t': '2662.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.449', 'sweep_vmp': '20.37', 'sweep_pmax': '36.992', 'sweep_voc': '22.925', 'vb_min_daily': '12.482', 'vb_max_daily': '13.748', 'ahc_daily': '1.9', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930244409 ms)
2025-02-18 14:19:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:18:25.400163+00:00', 'runtime': '930180.001', 'adc_vb_f': '13.739', 'adc_va_f': '20.361', 'adc_vl_f': '13.73', 'adc_ic_f': '2.971', 'adc_il_f': '0.345', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.708', 'vb_ref': '14.266', 'ahc_r': '2662.2', 'ahc_t': '2662.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.811', 'sweep_vmp': '20.37', 'sweep_pmax': '36.992', 'sweep_voc': '22.925', 'vb_min_daily': '12.482', 'vb_max_daily': '13.739', 'ahc_daily': '1.8', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930244410 ms)
2025-02-18 14:20:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930304413 ms)
2025-02-18 14:20:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930304413 ms)
2025-02-18 14:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:20:25.400185+00:00', 'runtime': '930300.001', 'adc_vb_f': '13.815', 'adc_va_f': '20.599', 'adc_vl_f': '13.806', 'adc_ic_f': '3.143', 'adc_il_f': '0.365', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.791', 'vb_ref': '14.266', 'ahc_r': '2662.3', 'ahc_t': '2662.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.408', 'sweep_vmp': '20.609', 'sweep_pmax': '43.68', 'sweep_voc': '23.059', 'vb_min_daily': '12.482', 'vb_max_daily': '13.815', 'ahc_daily': '1.9', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930304414 ms)
2025-02-18 14:20:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:19:25.400153+00:00', 'runtime': '930240.001', 'adc_vb_f': '13.748', 'adc_va_f': '20.361', 'adc_vl_f': '13.73', 'adc_ic_f': '2.942', 'adc_il_f': '0.348', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.739', 'vb_ref': '14.266', 'ahc_r': '2662.3', 'ahc_t': '2662.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '40.449', 'sweep_vmp': '20.37', 'sweep_pmax': '36.992', 'sweep_voc': '22.925', 'vb_min_daily': '12.482', 'vb_max_daily': '13.748', 'ahc_daily': '1.9', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930304415 ms)
2025-02-18 14:20:35.588 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+930313407 ms)
2025-02-18 14:20:35.588 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+930313408 ms)
2025-02-18 14:20:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:20:25.400336+00:00'} (T+930313410 ms)
2025-02-18 14:21:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930364418 ms)
2025-02-18 14:21:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930364418 ms)
2025-02-18 14:21:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:21:25.400197+00:00', 'runtime': '930360.001', 'adc_vb_f': '13.843', 'adc_va_f': '20.599', 'adc_vl_f': '13.809', 'adc_ic_f': '3.107', 'adc_il_f': '0.358', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.824', 'vb_ref': '14.266', 'ahc_r': '2662.4', 'ahc_t': '2662.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '42.955', 'sweep_vmp': '20.609', 'sweep_pmax': '43.68', 'sweep_voc': '23.059', 'vb_min_daily': '12.482', 'vb_max_daily': '13.852', 'ahc_daily': '2.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930364419 ms)
2025-02-18 14:21:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:20:25.400185+00:00', 'runtime': '930300.001', 'adc_vb_f': '13.815', 'adc_va_f': '20.599', 'adc_vl_f': '13.806', 'adc_ic_f': '3.143', 'adc_il_f': '0.365', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.791', 'vb_ref': '14.266', 'ahc_r': '2662.3', 'ahc_t': '2662.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.3', 'ahl_t': '1599.3', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.408', 'sweep_vmp': '20.609', 'sweep_pmax': '43.68', 'sweep_voc': '23.059', 'vb_min_daily': '12.482', 'vb_max_daily': '13.815', 'ahc_daily': '1.9', 'ahl_daily': '6.0', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930364420 ms)
2025-02-18 14:22:10.258 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930408078 ms)
2025-02-18 14:22:10.259 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930408078 ms)
2025-02-18 14:22:10.259 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 44974)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930408078 ms)
2025-02-18 14:22:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930424409 ms)
2025-02-18 14:22:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930424409 ms)
2025-02-18 14:22:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:22:25.400173+00:00', 'runtime': '930420.001', 'adc_vb_f': '13.925', 'adc_va_f': '20.596', 'adc_vl_f': '13.882', 'adc_ic_f': '3.208', 'adc_il_f': '0.353', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.925', 'vb_ref': '14.266', 'ahc_r': '2662.4', 'ahc_t': '2662.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '44.616', 'sweep_vmp': '20.609', 'sweep_pmax': '43.68', 'sweep_voc': '23.059', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930424410 ms)
2025-02-18 14:22:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:21:25.400197+00:00', 'runtime': '930360.001', 'adc_vb_f': '13.843', 'adc_va_f': '20.599', 'adc_vl_f': '13.809', 'adc_ic_f': '3.107', 'adc_il_f': '0.358', 't_hs': '9', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.824', 'vb_ref': '14.266', 'ahc_r': '2662.4', 'ahc_t': '2662.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '42.955', 'sweep_vmp': '20.609', 'sweep_pmax': '43.68', 'sweep_voc': '23.059', 'vb_min_daily': '12.482', 'vb_max_daily': '13.852', 'ahc_daily': '2.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930424411 ms)
2025-02-18 14:23:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930484413 ms)
2025-02-18 14:23:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930484414 ms)
2025-02-18 14:23:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:23:25.400204+00:00', 'runtime': '930480.001', 'adc_vb_f': '13.922', 'adc_va_f': '20.599', 'adc_vl_f': '13.882', 'adc_ic_f': '3.148', 'adc_il_f': '0.355', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.901', 'vb_ref': '14.237', 'ahc_r': '2662.5', 'ahc_t': '2662.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.816', 'sweep_vmp': '20.612', 'sweep_pmax': '42.261', 'sweep_voc': '22.977', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930484415 ms)
2025-02-18 14:23:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:22:25.400173+00:00', 'runtime': '930420.001', 'adc_vb_f': '13.925', 'adc_va_f': '20.596', 'adc_vl_f': '13.882', 'adc_ic_f': '3.208', 'adc_il_f': '0.353', 't_hs': '10', 't_batt': '10', 't_amb': '10', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.925', 'vb_ref': '14.266', 'ahc_r': '2662.4', 'ahc_t': '2662.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '44.616', 'sweep_vmp': '20.609', 'sweep_pmax': '43.68', 'sweep_voc': '23.059', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930484415 ms)
2025-02-18 14:23:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+930493420 ms)
2025-02-18 14:23:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+930493421 ms)
2025-02-18 14:23:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:23:25.400340+00:00'} (T+930493425 ms)
2025-02-18 14:24:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930544402 ms)
2025-02-18 14:24:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930544403 ms)
2025-02-18 14:24:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:24:25.400191+00:00', 'runtime': '930540.001', 'adc_vb_f': '13.956', 'adc_va_f': '20.599', 'adc_vl_f': '13.919', 'adc_ic_f': '3.184', 'adc_il_f': '0.285', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.91', 'vb_ref': '14.237', 'ahc_r': '2662.5', 'ahc_t': '2662.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '44.39', 'sweep_vmp': '20.612', 'sweep_pmax': '42.261', 'sweep_voc': '22.977', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930544404 ms)
2025-02-18 14:24:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:23:25.400204+00:00', 'runtime': '930480.001', 'adc_vb_f': '13.922', 'adc_va_f': '20.599', 'adc_vl_f': '13.882', 'adc_ic_f': '3.148', 'adc_il_f': '0.355', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.901', 'vb_ref': '14.237', 'ahc_r': '2662.5', 'ahc_t': '2662.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.816', 'sweep_vmp': '20.612', 'sweep_pmax': '42.261', 'sweep_voc': '22.977', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930544405 ms)
2025-02-18 14:25:02.556 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930580375 ms)
2025-02-18 14:25:02.556 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930580376 ms)
2025-02-18 14:25:02.557 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45158)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930580376 ms)
2025-02-18 14:25:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930604407 ms)
2025-02-18 14:25:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930604408 ms)
2025-02-18 14:25:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:25:25.400177+00:00', 'runtime': '930600.001', 'adc_vb_f': '13.852', 'adc_va_f': '20.599', 'adc_vl_f': '13.843', 'adc_ic_f': '2.771', 'adc_il_f': '0.295', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.879', 'vb_ref': '14.237', 'ahc_r': '2662.6', 'ahc_t': '2662.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '38.305', 'sweep_vmp': '20.612', 'sweep_pmax': '42.261', 'sweep_voc': '22.977', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930604409 ms)
2025-02-18 14:25:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:24:25.400191+00:00', 'runtime': '930540.001', 'adc_vb_f': '13.956', 'adc_va_f': '20.599', 'adc_vl_f': '13.919', 'adc_ic_f': '3.184', 'adc_il_f': '0.285', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.91', 'vb_ref': '14.237', 'ahc_r': '2662.5', 'ahc_t': '2662.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '44.39', 'sweep_vmp': '20.612', 'sweep_pmax': '42.261', 'sweep_voc': '22.977', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930604410 ms)
2025-02-18 14:25:59.715 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930637534 ms)
2025-02-18 14:25:59.715 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930637535 ms)
2025-02-18 14:25:59.716 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45228)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930637535 ms)
2025-02-18 14:26:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930664414 ms)
2025-02-18 14:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930664415 ms)
2025-02-18 14:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:26:25.400175+00:00', 'runtime': '930660.001', 'adc_vb_f': '13.898', 'adc_va_f': '20.456', 'adc_vl_f': '13.882', 'adc_ic_f': '3.003', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.879', 'vb_ref': '14.237', 'ahc_r': '2662.6', 'ahc_t': '2662.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '41.672', 'sweep_vmp': '20.468', 'sweep_pmax': '41.189', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930664416 ms)
2025-02-18 14:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:25:25.400177+00:00', 'runtime': '930600.001', 'adc_vb_f': '13.852', 'adc_va_f': '20.599', 'adc_vl_f': '13.843', 'adc_ic_f': '2.771', 'adc_il_f': '0.295', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.879', 'vb_ref': '14.237', 'ahc_r': '2662.6', 'ahc_t': '2662.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '38.305', 'sweep_vmp': '20.612', 'sweep_pmax': '42.261', 'sweep_voc': '22.977', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930664417 ms)
2025-02-18 14:26:59.871 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930697690 ms)
2025-02-18 14:26:59.871 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930697691 ms)
2025-02-18 14:26:59.872 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45294)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930697691 ms)
2025-02-18 14:27:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930724405 ms)
2025-02-18 14:27:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930724406 ms)
2025-02-18 14:27:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:27:25.400186+00:00', 'runtime': '930720.001', 'adc_vb_f': '14.081', 'adc_va_f': '20.459', 'adc_vl_f': '14.072', 'adc_ic_f': '3.271', 'adc_il_f': '0.333', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.035', 'vb_ref': '14.237', 'ahc_r': '2662.7', 'ahc_t': '2662.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '46.02', 'sweep_vmp': '20.468', 'sweep_pmax': '41.189', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '14.114', 'ahc_daily': '2.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930724407 ms)
2025-02-18 14:27:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:26:25.400175+00:00', 'runtime': '930660.001', 'adc_vb_f': '13.898', 'adc_va_f': '20.456', 'adc_vl_f': '13.882', 'adc_ic_f': '3.003', 'adc_il_f': '0.321', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '13.879', 'vb_ref': '14.237', 'ahc_r': '2662.6', 'ahc_t': '2662.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '41.672', 'sweep_vmp': '20.468', 'sweep_pmax': '41.189', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '13.968', 'ahc_daily': '2.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930724407 ms)
2025-02-18 14:28:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930784415 ms)
2025-02-18 14:28:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930784415 ms)
2025-02-18 14:28:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:28:25.400270+00:00', 'runtime': '930780.001', 'adc_vb_f': '14.035', 'adc_va_f': '20.456', 'adc_vl_f': '13.995', 'adc_ic_f': '3.099', 'adc_il_f': '0.348', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.069', 'vb_ref': '14.237', 'ahc_r': '2662.8', 'ahc_t': '2662.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.484', 'sweep_vmp': '20.468', 'sweep_pmax': '41.189', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '14.124', 'ahc_daily': '2.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930784416 ms)
2025-02-18 14:28:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:27:25.400186+00:00', 'runtime': '930720.001', 'adc_vb_f': '14.081', 'adc_va_f': '20.459', 'adc_vl_f': '14.072', 'adc_ic_f': '3.271', 'adc_il_f': '0.333', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.035', 'vb_ref': '14.237', 'ahc_r': '2662.7', 'ahc_t': '2662.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '46.02', 'sweep_vmp': '20.468', 'sweep_pmax': '41.189', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '14.114', 'ahc_daily': '2.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930784419 ms)
2025-02-18 14:29:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930844406 ms)
2025-02-18 14:29:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930844407 ms)
2025-02-18 14:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:29:25.400178+00:00', 'runtime': '930840.001', 'adc_vb_f': '14.691', 'adc_va_f': '20.599', 'adc_vl_f': '14.645', 'adc_ic_f': '3.662', 'adc_il_f': '0.283', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '14.237', 'ahc_r': '2662.8', 'ahc_t': '2662.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '53.796', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.691', 'ahc_daily': '2.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930844408 ms)
2025-02-18 14:29:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:28:25.400270+00:00', 'runtime': '930780.001', 'adc_vb_f': '14.035', 'adc_va_f': '20.456', 'adc_vl_f': '13.995', 'adc_ic_f': '3.099', 'adc_il_f': '0.348', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.069', 'vb_ref': '14.237', 'ahc_r': '2662.8', 'ahc_t': '2662.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '43.484', 'sweep_vmp': '20.468', 'sweep_pmax': '41.189', 'sweep_voc': '22.949', 'vb_min_daily': '12.482', 'vb_max_daily': '14.124', 'ahc_daily': '2.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930844410 ms)
2025-02-18 14:30:06.177 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+930883996 ms)
2025-02-18 14:30:06.177 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+930883996 ms)
2025-02-18 14:30:06.178 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45490)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+930883997 ms)
2025-02-18 14:30:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930904412 ms)
2025-02-18 14:30:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930904413 ms)
2025-02-18 14:30:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:30:25.400696+00:00', 'runtime': '930900.001', 'adc_vb_f': '14.725', 'adc_va_f': '21.527', 'adc_vl_f': '14.685', 'adc_ic_f': '3.365', 'adc_il_f': '0.278', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.237', 'ahc_r': '2662.9', 'ahc_t': '2662.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.641', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930904416 ms)
2025-02-18 14:30:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:29:25.400178+00:00', 'runtime': '930840.001', 'adc_vb_f': '14.691', 'adc_va_f': '20.599', 'adc_vl_f': '14.645', 'adc_ic_f': '3.662', 'adc_il_f': '0.283', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '5', 'array_fault': '0', 'vb_f': '14.349', 'vb_ref': '14.237', 'ahc_r': '2662.8', 'ahc_t': '2662.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '6', 'power_out': '53.796', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.691', 'ahc_daily': '2.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930904417 ms)
2025-02-18 14:31:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+930964421 ms)
2025-02-18 14:31:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+930964421 ms)
2025-02-18 14:31:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:31:25.402772+00:00', 'runtime': '930960.003', 'adc_vb_f': '14.728', 'adc_va_f': '21.671', 'adc_vl_f': '14.722', 'adc_ic_f': '3.397', 'adc_il_f': '0.29', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.725', 'vb_ref': '14.237', 'ahc_r': '2662.9', 'ahc_t': '2662.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.022', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930964425 ms)
2025-02-18 14:31:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:30:25.400696+00:00', 'runtime': '930900.001', 'adc_vb_f': '14.725', 'adc_va_f': '21.527', 'adc_vl_f': '14.685', 'adc_ic_f': '3.365', 'adc_il_f': '0.278', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.237', 'ahc_r': '2662.9', 'ahc_t': '2662.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.641', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+930964426 ms)
2025-02-18 14:32:04.538 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931002358 ms)
2025-02-18 14:32:04.539 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931002358 ms)
2025-02-18 14:32:04.539 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45616)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931002358 ms)
2025-02-18 14:32:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931024412 ms)
2025-02-18 14:32:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931024412 ms)
2025-02-18 14:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:32:25.400749+00:00', 'runtime': '931020.001', 'adc_vb_f': '14.728', 'adc_va_f': '21.985', 'adc_vl_f': '14.722', 'adc_ic_f': '3.358', 'adc_il_f': '0.249', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.728', 'vb_ref': '14.237', 'ahc_r': '2663.0', 'ahc_t': '2663.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '49.418', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931024414 ms)
2025-02-18 14:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:31:25.402772+00:00', 'runtime': '930960.003', 'adc_vb_f': '14.728', 'adc_va_f': '21.671', 'adc_vl_f': '14.722', 'adc_ic_f': '3.397', 'adc_il_f': '0.29', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.725', 'vb_ref': '14.237', 'ahc_r': '2662.9', 'ahc_t': '2662.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '50.022', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931024414 ms)
2025-02-18 14:33:00.666 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931058485 ms)
2025-02-18 14:33:00.666 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931058486 ms)
2025-02-18 14:33:00.667 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45676)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931058486 ms)
2025-02-18 14:33:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931084421 ms)
2025-02-18 14:33:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931084421 ms)
2025-02-18 14:33:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:33:25.400259+00:00', 'runtime': '931080.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.878', 'adc_vl_f': '14.682', 'adc_ic_f': '3.273', 'adc_il_f': '0.283', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.722', 'vb_ref': '14.207', 'ahc_r': '2663.0', 'ahc_t': '2663.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.089', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931084423 ms)
2025-02-18 14:33:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:32:25.400749+00:00', 'runtime': '931020.001', 'adc_vb_f': '14.728', 'adc_va_f': '21.985', 'adc_vl_f': '14.722', 'adc_ic_f': '3.358', 'adc_il_f': '0.249', 't_hs': '10', 't_batt': '11', 't_amb': '11', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.728', 'vb_ref': '14.237', 'ahc_r': '2663.0', 'ahc_t': '2663.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '49.418', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931084423 ms)
2025-02-18 14:34:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931144415 ms)
2025-02-18 14:34:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931144415 ms)
2025-02-18 14:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:34:25.400170+00:00', 'runtime': '931140.001', 'adc_vb_f': '14.697', 'adc_va_f': '22.006', 'adc_vl_f': '14.682', 'adc_ic_f': '3.38', 'adc_il_f': '0.268', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.694', 'vb_ref': '14.207', 'ahc_r': '2663.1', 'ahc_t': '2663.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.361', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931144416 ms)
2025-02-18 14:34:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:33:25.400259+00:00', 'runtime': '931080.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.878', 'adc_vl_f': '14.682', 'adc_ic_f': '3.273', 'adc_il_f': '0.283', 't_hs': '10', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.722', 'vb_ref': '14.207', 'ahc_r': '2663.0', 'ahc_t': '2663.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.089', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931144417 ms)
2025-02-18 14:35:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931204410 ms)
2025-02-18 14:35:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931204410 ms)
2025-02-18 14:35:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:35:25.400243+00:00', 'runtime': '931200.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.308', 'adc_vl_f': '14.682', 'adc_ic_f': '3.157', 'adc_il_f': '0.292', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.1', 'ahc_t': '2663.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.383', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931204411 ms)
2025-02-18 14:35:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:34:25.400170+00:00', 'runtime': '931140.001', 'adc_vb_f': '14.697', 'adc_va_f': '22.006', 'adc_vl_f': '14.682', 'adc_ic_f': '3.38', 'adc_il_f': '0.268', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.694', 'vb_ref': '14.207', 'ahc_r': '2663.1', 'ahc_t': '2663.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '48.361', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931204412 ms)
2025-02-18 14:35:40.941 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931218760 ms)
2025-02-18 14:35:40.941 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931218761 ms)
2025-02-18 14:35:40.942 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45844)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931218761 ms)
2025-02-18 14:36:11.056 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931248875 ms)
2025-02-18 14:36:11.056 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931248875 ms)
2025-02-18 14:36:11.056 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 45888)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931248876 ms)
2025-02-18 14:36:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931264407 ms)
2025-02-18 14:36:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931264407 ms)
2025-02-18 14:36:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:36:25.400169+00:00', 'runtime': '931260.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.988', 'adc_vl_f': '14.682', 'adc_ic_f': '3.136', 'adc_il_f': '0.295', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.2', 'ahc_t': '2663.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.066', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931264409 ms)
2025-02-18 14:36:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:35:25.400243+00:00', 'runtime': '931200.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.308', 'adc_vl_f': '14.682', 'adc_ic_f': '3.157', 'adc_il_f': '0.292', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.1', 'ahc_t': '2663.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.383', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931264409 ms)
2025-02-18 14:36:35.595 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+931273414 ms)
2025-02-18 14:36:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+931273415 ms)
2025-02-18 14:36:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:36:25.400330+00:00'} (T+931273419 ms)
2025-02-18 14:37:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931324417 ms)
2025-02-18 14:37:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931324417 ms)
2025-02-18 14:37:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:37:25.400176+00:00', 'runtime': '931320.001', 'adc_vb_f': '14.691', 'adc_va_f': '20.898', 'adc_vl_f': '14.645', 'adc_ic_f': '3.104', 'adc_il_f': '0.287', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.2', 'ahc_t': '2663.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '45.567', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931324422 ms)
2025-02-18 14:37:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:36:25.400169+00:00', 'runtime': '931260.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.988', 'adc_vl_f': '14.682', 'adc_ic_f': '3.136', 'adc_il_f': '0.295', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.2', 'ahc_t': '2663.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.066', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931324422 ms)
2025-02-18 14:37:35.625 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+931333444 ms)
2025-02-18 14:37:35.625 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+931333445 ms)
2025-02-18 14:37:35.626 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:37:25.400207+00:00'} (T+931333446 ms)
2025-02-18 14:38:26.279 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931384098 ms)
2025-02-18 14:38:26.279 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931384099 ms)
2025-02-18 14:38:26.280 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46032)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931384099 ms)
2025-02-18 14:38:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931384410 ms)
2025-02-18 14:38:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931384411 ms)
2025-02-18 14:38:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:38:25.400189+00:00', 'runtime': '931380.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.744', 'adc_vl_f': '14.682', 'adc_ic_f': '3.058', 'adc_il_f': '0.283', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.3', 'ahc_t': '2663.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.888', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.9', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931384413 ms)
2025-02-18 14:38:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:37:25.400176+00:00', 'runtime': '931320.001', 'adc_vb_f': '14.691', 'adc_va_f': '20.898', 'adc_vl_f': '14.645', 'adc_ic_f': '3.104', 'adc_il_f': '0.287', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.2', 'ahc_t': '2663.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '45.567', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931384414 ms)
2025-02-18 14:39:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931444403 ms)
2025-02-18 14:39:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931444404 ms)
2025-02-18 14:39:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:39:25.400205+00:00', 'runtime': '931440.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.388', 'adc_vl_f': '14.682', 'adc_ic_f': '3.008', 'adc_il_f': '0.273', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.4', 'ahc_t': '2663.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.066', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931444408 ms)
2025-02-18 14:39:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:38:25.400189+00:00', 'runtime': '931380.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.744', 'adc_vl_f': '14.682', 'adc_ic_f': '3.058', 'adc_il_f': '0.283', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.3', 'ahc_t': '2663.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '44.888', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '2.9', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931444408 ms)
2025-02-18 14:39:44.378 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+931462197 ms)
2025-02-18 14:39:44.378 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+931462197 ms)
2025-02-18 14:39:44.378 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931462197 ms)
2025-02-18 14:40:08.487 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931486306 ms)
2025-02-18 14:40:08.487 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931486307 ms)
2025-02-18 14:40:08.488 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46142)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931486307 ms)
2025-02-18 14:40:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931504415 ms)
2025-02-18 14:40:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931504416 ms)
2025-02-18 14:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:40:25.400255+00:00', 'runtime': '931500.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.229', 'adc_vl_f': '14.682', 'adc_ic_f': '2.962', 'adc_il_f': '0.268', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.4', 'ahc_t': '2663.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.499', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931504419 ms)
2025-02-18 14:40:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:39:25.400205+00:00', 'runtime': '931440.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.388', 'adc_vl_f': '14.682', 'adc_ic_f': '3.008', 'adc_il_f': '0.273', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.4', 'ahc_t': '2663.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '46.066', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931504420 ms)
2025-02-18 14:41:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931564411 ms)
2025-02-18 14:41:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931564412 ms)
2025-02-18 14:41:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:41:25.400712+00:00', 'runtime': '931560.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.202', 'adc_vl_f': '14.682', 'adc_ic_f': '2.95', 'adc_il_f': '0.246', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.5', 'ahc_t': '2663.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.288', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931564413 ms)
2025-02-18 14:41:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:40:25.400255+00:00', 'runtime': '931500.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.229', 'adc_vl_f': '14.682', 'adc_ic_f': '2.962', 'adc_il_f': '0.268', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.4', 'ahc_t': '2663.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.499', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931564413 ms)
2025-02-18 14:42:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931624404 ms)
2025-02-18 14:42:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931624405 ms)
2025-02-18 14:42:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:42:25.400596+00:00', 'runtime': '931620.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.664', 'adc_vl_f': '14.658', 'adc_ic_f': '2.909', 'adc_il_f': '0.251', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.5', 'ahc_t': '2663.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.729', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931624406 ms)
2025-02-18 14:42:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:41:25.400712+00:00', 'runtime': '931560.001', 'adc_vb_f': '14.691', 'adc_va_f': '22.202', 'adc_vl_f': '14.682', 'adc_ic_f': '2.95', 'adc_il_f': '0.246', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.5', 'ahc_t': '2663.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '43.288', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931624407 ms)
2025-02-18 14:43:16.793 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931674613 ms)
2025-02-18 14:43:16.794 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931674613 ms)
2025-02-18 14:43:16.794 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46340)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931674613 ms)
2025-02-18 14:43:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931684414 ms)
2025-02-18 14:43:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931684415 ms)
2025-02-18 14:43:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:43:25.400291+00:00', 'runtime': '931680.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.237', 'adc_vl_f': '14.645', 'adc_ic_f': '2.909', 'adc_il_f': '0.266', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.682', 'vb_ref': '14.178', 'ahc_r': '2663.5', 'ahc_t': '2663.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.578', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931684416 ms)
2025-02-18 14:43:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:42:25.400596+00:00', 'runtime': '931620.001', 'adc_vb_f': '14.691', 'adc_va_f': '21.664', 'adc_vl_f': '14.658', 'adc_ic_f': '2.909', 'adc_il_f': '0.251', 't_hs': '11', 't_batt': '12', 't_amb': '12', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.691', 'vb_ref': '14.207', 'ahc_r': '2663.5', 'ahc_t': '2663.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.729', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931684416 ms)
2025-02-18 14:44:02.924 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931720744 ms)
2025-02-18 14:44:02.925 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931720744 ms)
2025-02-18 14:44:02.925 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46392)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931720744 ms)
2025-02-18 14:44:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931744406 ms)
2025-02-18 14:44:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931744407 ms)
2025-02-18 14:44:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:44:25.400751+00:00', 'runtime': '931740.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.954', 'adc_vl_f': '14.645', 'adc_ic_f': '2.884', 'adc_il_f': '0.283', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.6', 'ahc_t': '2663.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.261', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931744408 ms)
2025-02-18 14:44:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:43:25.400291+00:00', 'runtime': '931680.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.237', 'adc_vl_f': '14.645', 'adc_ic_f': '2.909', 'adc_il_f': '0.266', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.682', 'vb_ref': '14.178', 'ahc_r': '2663.5', 'ahc_t': '2663.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.578', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931744408 ms)
2025-02-18 14:44:46.075 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931763894 ms)
2025-02-18 14:44:46.075 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931763894 ms)
2025-02-18 14:44:46.076 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46436)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931763895 ms)
2025-02-18 14:45:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931804416 ms)
2025-02-18 14:45:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931804416 ms)
2025-02-18 14:45:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:45:25.400600+00:00', 'runtime': '931800.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.177', 'adc_vl_f': '14.645', 'adc_ic_f': '2.855', 'adc_il_f': '0.292', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2663.6', 'ahc_t': '2663.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.793', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931804417 ms)
2025-02-18 14:45:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:44:25.400751+00:00', 'runtime': '931740.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.954', 'adc_vl_f': '14.645', 'adc_ic_f': '2.884', 'adc_il_f': '0.283', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.6', 'ahc_t': '2663.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '42.261', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931804418 ms)
2025-02-18 14:46:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931864412 ms)
2025-02-18 14:46:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931864412 ms)
2025-02-18 14:46:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:46:25.400167+00:00', 'runtime': '931860.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.202', 'adc_vl_f': '14.645', 'adc_ic_f': '2.829', 'adc_il_f': '0.297', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.7', 'ahc_t': '2663.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.4', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931864413 ms)
2025-02-18 14:46:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:45:25.400600+00:00', 'runtime': '931800.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.177', 'adc_vl_f': '14.645', 'adc_ic_f': '2.855', 'adc_il_f': '0.292', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2663.6', 'ahc_t': '2663.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.793', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931864414 ms)
2025-02-18 14:47:23.362 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931921181 ms)
2025-02-18 14:47:23.362 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931921181 ms)
2025-02-18 14:47:23.363 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46612)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931921182 ms)
2025-02-18 14:47:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931924404 ms)
2025-02-18 14:47:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931924404 ms)
2025-02-18 14:47:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:47:25.400178+00:00', 'runtime': '931920.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.33', 'adc_vl_f': '14.645', 'adc_ic_f': '2.797', 'adc_il_f': '0.287', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.7', 'ahc_t': '2663.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.978', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931924405 ms)
2025-02-18 14:47:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:46:25.400167+00:00', 'runtime': '931860.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.202', 'adc_vl_f': '14.645', 'adc_ic_f': '2.829', 'adc_il_f': '0.297', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.7', 'ahc_t': '2663.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '41.4', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931924405 ms)
2025-02-18 14:47:41.446 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+931939265 ms)
2025-02-18 14:47:41.446 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+931939265 ms)
2025-02-18 14:47:41.446 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46632)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+931939266 ms)
2025-02-18 14:48:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+931984415 ms)
2025-02-18 14:48:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+931984416 ms)
2025-02-18 14:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:48:25.400182+00:00', 'runtime': '931980.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.202', 'adc_vl_f': '14.645', 'adc_ic_f': '2.781', 'adc_il_f': '0.266', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2663.8', 'ahc_t': '2663.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.691', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931984417 ms)
2025-02-18 14:48:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:47:25.400178+00:00', 'runtime': '931920.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.33', 'adc_vl_f': '14.645', 'adc_ic_f': '2.797', 'adc_il_f': '0.287', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.7', 'ahc_t': '2663.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.978', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+931984417 ms)
2025-02-18 14:49:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932044408 ms)
2025-02-18 14:49:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932044408 ms)
2025-02-18 14:49:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:49:25.400265+00:00', 'runtime': '932040.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.043', 'adc_vl_f': '14.645', 'adc_ic_f': '2.759', 'adc_il_f': '0.244', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2663.8', 'ahc_t': '2663.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.419', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932044409 ms)
2025-02-18 14:49:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:48:25.400182+00:00', 'runtime': '931980.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.202', 'adc_vl_f': '14.645', 'adc_ic_f': '2.781', 'adc_il_f': '0.266', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2663.8', 'ahc_t': '2663.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.691', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932044410 ms)
2025-02-18 14:50:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932104415 ms)
2025-02-18 14:50:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932104415 ms)
2025-02-18 14:50:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:50:25.400625+00:00', 'runtime': '932100.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.009', 'adc_vl_f': '14.645', 'adc_ic_f': '2.725', 'adc_il_f': '0.246', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.9', 'ahc_t': '2663.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.875', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932104420 ms)
2025-02-18 14:50:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:49:25.400265+00:00', 'runtime': '932040.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.043', 'adc_vl_f': '14.645', 'adc_ic_f': '2.759', 'adc_il_f': '0.244', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2663.8', 'ahc_t': '2663.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '40.419', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932104420 ms)
2025-02-18 14:50:48.757 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+932126576 ms)
2025-02-18 14:50:48.757 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+932126576 ms)
2025-02-18 14:50:48.757 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 46828)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+932126577 ms)
2025-02-18 14:51:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932164409 ms)
2025-02-18 14:51:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932164409 ms)
2025-02-18 14:51:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:51:25.400167+00:00', 'runtime': '932160.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.202', 'adc_vl_f': '14.645', 'adc_ic_f': '2.713', 'adc_il_f': '0.283', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.9', 'ahc_t': '2663.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.739', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932164414 ms)
2025-02-18 14:51:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:50:25.400625+00:00', 'runtime': '932100.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.009', 'adc_vl_f': '14.645', 'adc_ic_f': '2.725', 'adc_il_f': '0.246', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.9', 'ahc_t': '2663.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.875', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932164414 ms)
2025-02-18 14:51:35.652 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+932173471 ms)
2025-02-18 14:51:35.653 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+932173472 ms)
2025-02-18 14:51:35.658 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:51:25.405064+00:00'} (T+932173477 ms)
2025-02-18 14:52:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932224418 ms)
2025-02-18 14:52:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932224419 ms)
2025-02-18 14:52:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:52:25.402750+00:00', 'runtime': '932220.003', 'adc_vb_f': '14.651', 'adc_va_f': '21.988', 'adc_vl_f': '14.645', 'adc_ic_f': '2.674', 'adc_il_f': '0.297', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2664.0', 'ahc_t': '2664.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.181', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932224423 ms)
2025-02-18 14:52:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:51:25.400167+00:00', 'runtime': '932160.001', 'adc_vb_f': '14.651', 'adc_va_f': '22.202', 'adc_vl_f': '14.645', 'adc_ic_f': '2.713', 'adc_il_f': '0.283', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2663.9', 'ahc_t': '2663.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.739', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.5', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932224424 ms)
2025-02-18 14:52:35.599 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+932233418 ms)
2025-02-18 14:52:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+932233418 ms)
2025-02-18 14:52:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 14:52:25.400171+00:00'} (T+932233421 ms)
2025-02-18 14:53:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932284414 ms)
2025-02-18 14:53:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932284414 ms)
2025-02-18 14:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:53:25.400211+00:00', 'runtime': '932280.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.86', 'adc_vl_f': '14.645', 'adc_ic_f': '2.665', 'adc_il_f': '0.285', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2664.0', 'ahc_t': '2664.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.015', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932284415 ms)
2025-02-18 14:53:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:52:25.402750+00:00', 'runtime': '932220.003', 'adc_vb_f': '14.651', 'adc_va_f': '21.988', 'adc_vl_f': '14.645', 'adc_ic_f': '2.674', 'adc_il_f': '0.297', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.655', 'vb_ref': '14.178', 'ahc_r': '2664.0', 'ahc_t': '2664.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.181', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932284416 ms)
2025-02-18 14:53:30.966 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+932288785 ms)
2025-02-18 14:53:30.966 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+932288785 ms)
2025-02-18 14:53:30.967 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+932288786 ms)
2025-02-18 14:53:30.968 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+932288787 ms)
2025-02-18 14:53:30.968 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+932288787 ms)
2025-02-18 14:53:30.968 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+932288788 ms)
2025-02-18 14:53:30.973 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+932288792 ms)
2025-02-18 14:53:30.976 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+932288795 ms)
2025-02-18 14:53:30.980 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+932288799 ms)
2025-02-18 14:53:30.982 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+932288801 ms)
2025-02-18 14:53:30.985 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+932288805 ms)
2025-02-18 14:53:30.990 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+932288809 ms)
2025-02-18 14:54:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932344408 ms)
2025-02-18 14:54:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932344409 ms)
2025-02-18 14:54:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:54:25.400789+00:00', 'runtime': '932340.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.771', 'adc_vl_f': '14.645', 'adc_ic_f': '2.655', 'adc_il_f': '0.29', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2664.1', 'ahc_t': '2664.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.894', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932344412 ms)
2025-02-18 14:54:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:53:25.400211+00:00', 'runtime': '932280.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.86', 'adc_vl_f': '14.645', 'adc_ic_f': '2.665', 'adc_il_f': '0.285', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2664.0', 'ahc_t': '2664.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20145', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '39.015', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.6', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932344412 ms)
2025-02-18 14:55:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932404402 ms)
2025-02-18 14:55:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932404402 ms)
2025-02-18 14:55:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:55:25.400189+00:00', 'runtime': '932400.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.634', 'adc_vl_f': '14.645', 'adc_ic_f': '2.633', 'adc_il_f': '0.273', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.633', 'vb_ref': '14.148', 'ahc_r': '2664.1', 'ahc_t': '2664.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.652', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932404403 ms)
2025-02-18 14:55:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:54:25.400789+00:00', 'runtime': '932340.001', 'adc_vb_f': '14.651', 'adc_va_f': '21.771', 'adc_vl_f': '14.645', 'adc_ic_f': '2.655', 'adc_il_f': '0.29', 't_hs': '12', 't_batt': '13', 't_amb': '13', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.651', 'vb_ref': '14.178', 'ahc_r': '2664.1', 'ahc_t': '2664.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.894', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932404404 ms)
2025-02-18 14:56:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932464418 ms)
2025-02-18 14:56:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932464418 ms)
2025-02-18 14:56:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:56:25.400198+00:00', 'runtime': '932460.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.445', 'adc_vl_f': '14.639', 'adc_ic_f': '2.631', 'adc_il_f': '0.295', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.1', 'ahc_t': '2664.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.637', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932464419 ms)
2025-02-18 14:56:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:55:25.400189+00:00', 'runtime': '932400.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.634', 'adc_vl_f': '14.645', 'adc_ic_f': '2.633', 'adc_il_f': '0.273', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.633', 'vb_ref': '14.148', 'ahc_r': '2664.1', 'ahc_t': '2664.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.652', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932464420 ms)
2025-02-18 14:57:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932524409 ms)
2025-02-18 14:57:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932524409 ms)
2025-02-18 14:57:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:57:25.400255+00:00', 'runtime': '932520.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.014', 'adc_vl_f': '14.606', 'adc_ic_f': '2.585', 'adc_il_f': '0.29', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.2', 'ahc_t': '2664.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.837', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932524410 ms)
2025-02-18 14:57:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:56:25.400198+00:00', 'runtime': '932460.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.445', 'adc_vl_f': '14.639', 'adc_ic_f': '2.631', 'adc_il_f': '0.295', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.1', 'ahc_t': '2664.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '38.637', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.7', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932524411 ms)
2025-02-18 14:58:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932584408 ms)
2025-02-18 14:58:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932584408 ms)
2025-02-18 14:58:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:58:25.400176+00:00', 'runtime': '932580.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.67', 'adc_vl_f': '14.606', 'adc_ic_f': '2.553', 'adc_il_f': '0.256', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.2', 'ahc_t': '2664.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.414', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932584409 ms)
2025-02-18 14:58:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:57:25.400255+00:00', 'runtime': '932520.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.014', 'adc_vl_f': '14.606', 'adc_ic_f': '2.585', 'adc_il_f': '0.29', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.2', 'ahc_t': '2664.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.837', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932584410 ms)
2025-02-18 14:59:16.507 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+932634326 ms)
2025-02-18 14:59:16.507 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+932634327 ms)
2025-02-18 14:59:16.508 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47372)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+932634327 ms)
2025-02-18 14:59:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932644416 ms)
2025-02-18 14:59:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932644417 ms)
2025-02-18 14:59:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 14:59:25.400270+00:00', 'runtime': '932640.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.593', 'adc_vl_f': '14.606', 'adc_ic_f': '2.527', 'adc_il_f': '0.237', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.3', 'ahc_t': '2664.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.022', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.9', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932644418 ms)
2025-02-18 14:59:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:58:25.400176+00:00', 'runtime': '932580.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.67', 'adc_vl_f': '14.606', 'adc_ic_f': '2.553', 'adc_il_f': '0.256', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.2', 'ahc_t': '2664.2', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.414', 'sweep_vmp': '20.609', 'sweep_pmax': '48.829', 'sweep_voc': '23.08', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.8', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932644418 ms)
2025-02-18 15:00:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932704411 ms)
2025-02-18 15:00:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932704412 ms)
2025-02-18 15:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:00:25.400181+00:00', 'runtime': '932700.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.807', 'adc_vl_f': '14.606', 'adc_ic_f': '2.5', 'adc_il_f': '0.222', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.3', 'ahc_t': '2664.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.569', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.9', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932704413 ms)
2025-02-18 15:00:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 14:59:25.400270+00:00', 'runtime': '932640.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.593', 'adc_vl_f': '14.606', 'adc_ic_f': '2.527', 'adc_il_f': '0.237', 't_hs': '12', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.3', 'ahc_t': '2664.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '37.022', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.9', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932704413 ms)
2025-02-18 15:01:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932764407 ms)
2025-02-18 15:01:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932764408 ms)
2025-02-18 15:01:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:01:25.400161+00:00', 'runtime': '932760.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.926', 'adc_vl_f': '14.606', 'adc_ic_f': '2.469', 'adc_il_f': '0.225', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.4', 'ahc_t': '2664.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.055', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932764409 ms)
2025-02-18 15:01:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:00:25.400181+00:00', 'runtime': '932700.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.807', 'adc_vl_f': '14.606', 'adc_ic_f': '2.5', 'adc_il_f': '0.222', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.3', 'ahc_t': '2664.3', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.569', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '3.9', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932764409 ms)
2025-02-18 15:02:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932824404 ms)
2025-02-18 15:02:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932824405 ms)
2025-02-18 15:02:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:02:25.400258+00:00', 'runtime': '932820.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.837', 'adc_vl_f': '14.606', 'adc_ic_f': '2.481', 'adc_il_f': '0.254', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.4', 'ahc_t': '2664.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.539', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932824406 ms)
2025-02-18 15:02:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:01:25.400161+00:00', 'runtime': '932760.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.926', 'adc_vl_f': '14.606', 'adc_ic_f': '2.469', 'adc_il_f': '0.225', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.4', 'ahc_t': '2664.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.055', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932824406 ms)
2025-02-18 15:03:06.866 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+932864685 ms)
2025-02-18 15:03:06.866 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+932864686 ms)
2025-02-18 15:03:06.867 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47616)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+932864686 ms)
2025-02-18 15:03:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932884410 ms)
2025-02-18 15:03:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932884411 ms)
2025-02-18 15:03:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:03:25.400170+00:00', 'runtime': '932880.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.602', 'adc_vl_f': '14.606', 'adc_ic_f': '2.467', 'adc_il_f': '0.263', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.4', 'ahc_t': '2664.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.935', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932884412 ms)
2025-02-18 15:03:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:02:25.400258+00:00', 'runtime': '932820.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.837', 'adc_vl_f': '14.606', 'adc_ic_f': '2.481', 'adc_il_f': '0.254', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.4', 'ahc_t': '2664.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '36.539', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932884412 ms)
2025-02-18 15:04:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+932944406 ms)
2025-02-18 15:04:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+932944406 ms)
2025-02-18 15:04:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:04:25.400179+00:00', 'runtime': '932940.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.593', 'adc_vl_f': '14.606', 'adc_ic_f': '2.442', 'adc_il_f': '0.268', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.5', 'ahc_t': '2664.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.648', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932944407 ms)
2025-02-18 15:04:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:03:25.400170+00:00', 'runtime': '932880.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.602', 'adc_vl_f': '14.606', 'adc_ic_f': '2.467', 'adc_il_f': '0.263', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.4', 'ahc_t': '2664.4', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.935', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.0', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+932944408 ms)
2025-02-18 15:05:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933004417 ms)
2025-02-18 15:05:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933004418 ms)
2025-02-18 15:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:05:25.400188+00:00', 'runtime': '933000.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.651', 'adc_vl_f': '14.606', 'adc_ic_f': '2.433', 'adc_il_f': '0.283', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.5', 'ahc_t': '2664.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.693', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933004419 ms)
2025-02-18 15:05:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:04:25.400179+00:00', 'runtime': '932940.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.593', 'adc_vl_f': '14.606', 'adc_ic_f': '2.442', 'adc_il_f': '0.268', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.5', 'ahc_t': '2664.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.648', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933004419 ms)
2025-02-18 15:05:35.667 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933013486 ms)
2025-02-18 15:05:35.668 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933013487 ms)
2025-02-18 15:05:35.668 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:05:25.400336+00:00'} (T+933013488 ms)
2025-02-18 15:05:59.157 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+933036977 ms)
2025-02-18 15:05:59.158 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+933036977 ms)
2025-02-18 15:05:59.158 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47808)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+933036977 ms)
2025-02-18 15:06:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933064413 ms)
2025-02-18 15:06:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933064414 ms)
2025-02-18 15:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:06:25.402756+00:00', 'runtime': '933060.003', 'adc_vb_f': '14.627', 'adc_va_f': '20.782', 'adc_vl_f': '14.606', 'adc_ic_f': '2.409', 'adc_il_f': '0.283', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.6', 'ahc_t': '2664.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.723', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933064415 ms)
2025-02-18 15:06:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:05:25.400188+00:00', 'runtime': '933000.001', 'adc_vb_f': '14.63', 'adc_va_f': '20.651', 'adc_vl_f': '14.606', 'adc_ic_f': '2.433', 'adc_il_f': '0.283', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.5', 'ahc_t': '2664.5', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.693', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.1', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933064415 ms)
2025-02-18 15:06:35.723 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933073542 ms)
2025-02-18 15:06:35.724 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933073543 ms)
2025-02-18 15:06:35.725 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:06:25.400144+00:00'} (T+933073544 ms)
2025-02-18 15:07:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933124409 ms)
2025-02-18 15:07:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933124409 ms)
2025-02-18 15:07:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:07:25.402707+00:00', 'runtime': '933120.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.011', 'adc_vl_f': '14.615', 'adc_ic_f': '2.387', 'adc_il_f': '0.28', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.6', 'ahc_t': '2664.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.149', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933124410 ms)
2025-02-18 15:07:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:06:25.402756+00:00', 'runtime': '933060.003', 'adc_vb_f': '14.627', 'adc_va_f': '20.782', 'adc_vl_f': '14.606', 'adc_ic_f': '2.409', 'adc_il_f': '0.283', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.6', 'ahc_t': '2664.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.723', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933124411 ms)
2025-02-18 15:07:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933133420 ms)
2025-02-18 15:07:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933133421 ms)
2025-02-18 15:07:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:07:25.400079+00:00'} (T+933133422 ms)
2025-02-18 15:08:23.414 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+933181234 ms)
2025-02-18 15:08:23.415 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+933181234 ms)
2025-02-18 15:08:23.415 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 47962)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+933181234 ms)
2025-02-18 15:08:26.600 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933184419 ms)
2025-02-18 15:08:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933184419 ms)
2025-02-18 15:08:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:08:25.401129+00:00', 'runtime': '933180.002', 'adc_vb_f': '14.627', 'adc_va_f': '21.35', 'adc_vl_f': '14.606', 'adc_ic_f': '2.203', 'adc_il_f': '0.321', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.6', 'ahc_t': '2664.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '33.021', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933184421 ms)
2025-02-18 15:08:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:07:25.402707+00:00', 'runtime': '933120.003', 'adc_vb_f': '14.63', 'adc_va_f': '21.011', 'adc_vl_f': '14.615', 'adc_ic_f': '2.387', 'adc_il_f': '0.28', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.6', 'ahc_t': '2664.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '35.149', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933184422 ms)
2025-02-18 15:08:35.612 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933193432 ms)
2025-02-18 15:08:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933193432 ms)
2025-02-18 15:08:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:08:25.400174+00:00'} (T+933193433 ms)
2025-02-18 15:09:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933244416 ms)
2025-02-18 15:09:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933244416 ms)
2025-02-18 15:09:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:09:25.400160+00:00', 'runtime': '933240.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.475', 'adc_vl_f': '14.606', 'adc_ic_f': '2.184', 'adc_il_f': '0.316', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.7', 'ahc_t': '2664.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.492', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933244417 ms)
2025-02-18 15:09:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:08:25.401129+00:00', 'runtime': '933180.002', 'adc_vb_f': '14.627', 'adc_va_f': '21.35', 'adc_vl_f': '14.606', 'adc_ic_f': '2.203', 'adc_il_f': '0.321', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.6', 'ahc_t': '2664.6', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '33.021', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.2', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933244418 ms)
2025-02-18 15:10:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933304413 ms)
2025-02-18 15:10:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933304414 ms)
2025-02-18 15:10:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:10:25.400168+00:00', 'runtime': '933300.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.606', 'adc_vl_f': '14.606', 'adc_ic_f': '2.177', 'adc_il_f': '0.345', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2664.7', 'ahc_t': '2664.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.069', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933304415 ms)
2025-02-18 15:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:09:25.400160+00:00', 'runtime': '933240.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.475', 'adc_vl_f': '14.606', 'adc_ic_f': '2.184', 'adc_il_f': '0.316', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.7', 'ahc_t': '2664.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.492', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933304415 ms)
2025-02-18 15:10:33.672 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+933311491 ms)
2025-02-18 15:10:33.672 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+933311491 ms)
2025-02-18 15:10:33.672 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48102)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+933311492 ms)
2025-02-18 15:11:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933364408 ms)
2025-02-18 15:11:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933364409 ms)
2025-02-18 15:11:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:11:25.400534+00:00', 'runtime': '933360.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.768', 'adc_vl_f': '14.606', 'adc_ic_f': '2.148', 'adc_il_f': '0.331', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.8', 'ahc_t': '2664.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.843', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933364411 ms)
2025-02-18 15:11:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:10:25.400168+00:00', 'runtime': '933300.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.606', 'adc_vl_f': '14.606', 'adc_ic_f': '2.177', 'adc_il_f': '0.345', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2664.7', 'ahc_t': '2664.7', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '32.069', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.3', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933364411 ms)
2025-02-18 15:12:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933424420 ms)
2025-02-18 15:12:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933424420 ms)
2025-02-18 15:12:26.603 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:12:25.400171+00:00', 'runtime': '933420.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.945', 'adc_vl_f': '14.606', 'adc_ic_f': '2.145', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2664.8', 'ahc_t': '2664.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.647', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.4', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933424422 ms)
2025-02-18 15:12:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:11:25.400534+00:00', 'runtime': '933360.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.768', 'adc_vl_f': '14.606', 'adc_ic_f': '2.148', 'adc_il_f': '0.331', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.8', 'ahc_t': '2664.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.4', 'ahl_t': '1599.4', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.843', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.4', 'ahl_daily': '6.1', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933424423 ms)
2025-02-18 15:12:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933433433 ms)
2025-02-18 15:12:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933433434 ms)
2025-02-18 15:12:35.616 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:12:25.413053+00:00'} (T+933433435 ms)
2025-02-18 15:13:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933484419 ms)
2025-02-18 15:13:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933484419 ms)
2025-02-18 15:13:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:13:25.400236+00:00', 'runtime': '933480.001', 'adc_vb_f': '14.63', 'adc_va_f': '22.089', 'adc_vl_f': '14.606', 'adc_ic_f': '2.138', 'adc_il_f': '0.343', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.8', 'ahc_t': '2664.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.375', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.4', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933484420 ms)
2025-02-18 15:13:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:12:25.400171+00:00', 'runtime': '933420.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.945', 'adc_vl_f': '14.606', 'adc_ic_f': '2.145', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2664.8', 'ahc_t': '2664.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.647', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.4', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933484421 ms)
2025-02-18 15:13:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933493432 ms)
2025-02-18 15:13:35.613 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933493433 ms)
2025-02-18 15:13:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:13:25.405145+00:00'} (T+933493434 ms)
2025-02-18 15:14:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933544414 ms)
2025-02-18 15:14:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933544414 ms)
2025-02-18 15:14:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:14:25.400153+00:00', 'runtime': '933540.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.095', 'adc_vl_f': '14.606', 'adc_ic_f': '2.119', 'adc_il_f': '0.365', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.9', 'ahc_t': '2664.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.058', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933544418 ms)
2025-02-18 15:14:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:13:25.400236+00:00', 'runtime': '933480.001', 'adc_vb_f': '14.63', 'adc_va_f': '22.089', 'adc_vl_f': '14.606', 'adc_ic_f': '2.138', 'adc_il_f': '0.343', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.8', 'ahc_t': '2664.8', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.375', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.4', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933544418 ms)
2025-02-18 15:14:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933553425 ms)
2025-02-18 15:14:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933553426 ms)
2025-02-18 15:14:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:14:25.413123+00:00'} (T+933553427 ms)
2025-02-18 15:15:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933604408 ms)
2025-02-18 15:15:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933604408 ms)
2025-02-18 15:15:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:15:25.400161+00:00', 'runtime': '933600.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.095', 'adc_vl_f': '14.606', 'adc_ic_f': '2.119', 'adc_il_f': '0.367', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.9', 'ahc_t': '2664.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.816', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933604409 ms)
2025-02-18 15:15:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:14:25.400153+00:00', 'runtime': '933540.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.095', 'adc_vl_f': '14.606', 'adc_ic_f': '2.119', 'adc_il_f': '0.365', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.9', 'ahc_t': '2664.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '31.058', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933604410 ms)
2025-02-18 15:15:35.586 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933613405 ms)
2025-02-18 15:15:35.586 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933613406 ms)
2025-02-18 15:15:35.587 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:15:25.400202+00:00'} (T+933613407 ms)
2025-02-18 15:15:46.136 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+933623956 ms)
2025-02-18 15:15:46.137 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+933623956 ms)
2025-02-18 15:15:46.137 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48442)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+933623956 ms)
2025-02-18 15:16:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933664407 ms)
2025-02-18 15:16:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933664407 ms)
2025-02-18 15:16:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:16:25.400212+00:00', 'runtime': '933660.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.305', 'adc_vl_f': '14.606', 'adc_ic_f': '2.082', 'adc_il_f': '0.36', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.9', 'ahc_t': '2664.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.665', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933664408 ms)
2025-02-18 15:16:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:15:25.400161+00:00', 'runtime': '933600.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.095', 'adc_vl_f': '14.606', 'adc_ic_f': '2.119', 'adc_il_f': '0.367', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.9', 'ahc_t': '2664.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.816', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933664409 ms)
2025-02-18 15:16:35.704 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933673523 ms)
2025-02-18 15:16:35.705 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933673524 ms)
2025-02-18 15:16:35.706 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:16:25.400240+00:00'} (T+933673525 ms)
2025-02-18 15:17:26.584 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933724403 ms)
2025-02-18 15:17:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933724404 ms)
2025-02-18 15:17:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:17:25.400191+00:00', 'runtime': '933720.001', 'adc_vb_f': '14.63', 'adc_va_f': '22.388', 'adc_vl_f': '14.606', 'adc_ic_f': '2.063', 'adc_il_f': '0.353', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.0', 'ahc_t': '2665.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.348', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933724405 ms)
2025-02-18 15:17:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:16:25.400212+00:00', 'runtime': '933660.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.305', 'adc_vl_f': '14.606', 'adc_ic_f': '2.082', 'adc_il_f': '0.36', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2664.9', 'ahc_t': '2664.9', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.665', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.5', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933724406 ms)
2025-02-18 15:17:35.696 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933733515 ms)
2025-02-18 15:17:35.697 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933733516 ms)
2025-02-18 15:17:35.698 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:17:25.400112+00:00'} (T+933733517 ms)
2025-02-18 15:18:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933784414 ms)
2025-02-18 15:18:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933784414 ms)
2025-02-18 15:18:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:18:25.400181+00:00', 'runtime': '933780.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.15', 'adc_vl_f': '14.606', 'adc_ic_f': '2.003', 'adc_il_f': '0.333', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.0', 'ahc_t': '2665.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.348', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933784415 ms)
2025-02-18 15:18:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:17:25.400191+00:00', 'runtime': '933720.001', 'adc_vb_f': '14.63', 'adc_va_f': '22.388', 'adc_vl_f': '14.606', 'adc_ic_f': '2.063', 'adc_il_f': '0.353', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.0', 'ahc_t': '2665.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.348', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933784416 ms)
2025-02-18 15:18:35.583 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933793403 ms)
2025-02-18 15:18:35.584 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933793403 ms)
2025-02-18 15:18:35.585 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:18:25.400116+00:00'} (T+933793404 ms)
2025-02-18 15:19:03.484 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+933821304 ms)
2025-02-18 15:19:03.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+933821304 ms)
2025-02-18 15:19:03.485 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48656)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+933821304 ms)
2025-02-18 15:19:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933844411 ms)
2025-02-18 15:19:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933844412 ms)
2025-02-18 15:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:19:25.400198+00:00', 'runtime': '933840.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.988', 'adc_vl_f': '14.606', 'adc_ic_f': '1.986', 'adc_il_f': '0.312', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.0', 'ahc_t': '2665.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.925', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933844413 ms)
2025-02-18 15:19:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:18:25.400181+00:00', 'runtime': '933780.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.15', 'adc_vl_f': '14.606', 'adc_ic_f': '2.003', 'adc_il_f': '0.333', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.0', 'ahc_t': '2665.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '30.348', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933844413 ms)
2025-02-18 15:20:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933904405 ms)
2025-02-18 15:20:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933904405 ms)
2025-02-18 15:20:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:20:25.400196+00:00', 'runtime': '933900.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.875', 'adc_vl_f': '14.606', 'adc_ic_f': '1.969', 'adc_il_f': '0.312', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.1', 'ahc_t': '2665.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.536', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933904406 ms)
2025-02-18 15:20:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:19:25.400198+00:00', 'runtime': '933840.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.988', 'adc_vl_f': '14.606', 'adc_ic_f': '1.986', 'adc_il_f': '0.312', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.0', 'ahc_t': '2665.0', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.925', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.6', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933904407 ms)
2025-02-18 15:21:03.694 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+933941513 ms)
2025-02-18 15:21:03.694 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+933941514 ms)
2025-02-18 15:21:03.695 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 48784)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+933941514 ms)
2025-02-18 15:21:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+933964416 ms)
2025-02-18 15:21:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+933964417 ms)
2025-02-18 15:21:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:21:25.400205+00:00', 'runtime': '933960.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.765', 'adc_vl_f': '14.606', 'adc_ic_f': '1.959', 'adc_il_f': '0.324', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.1', 'ahc_t': '2665.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.261', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933964418 ms)
2025-02-18 15:21:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:20:25.400196+00:00', 'runtime': '933900.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.875', 'adc_vl_f': '14.606', 'adc_ic_f': '1.969', 'adc_il_f': '0.312', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.1', 'ahc_t': '2665.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.536', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+933964418 ms)
2025-02-18 15:21:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933973420 ms)
2025-02-18 15:21:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+933973421 ms)
2025-02-18 15:21:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:21:25.400338+00:00'} (T+933973422 ms)
2025-02-18 15:22:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934024412 ms)
2025-02-18 15:22:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934024412 ms)
2025-02-18 15:22:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:22:25.400214+00:00', 'runtime': '934020.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.725', 'adc_vl_f': '14.606', 'adc_ic_f': '1.945', 'adc_il_f': '0.324', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.1', 'ahc_t': '2665.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.186', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934024416 ms)
2025-02-18 15:22:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:21:25.400205+00:00', 'runtime': '933960.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.765', 'adc_vl_f': '14.606', 'adc_ic_f': '1.959', 'adc_il_f': '0.324', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.1', 'ahc_t': '2665.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.261', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934024417 ms)
2025-02-18 15:22:35.423 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934033243 ms)
2025-02-18 15:22:35.424 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934033243 ms)
2025-02-18 15:22:35.425 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:22:25.400226+00:00'} (T+934033244 ms)
2025-02-18 15:23:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934084409 ms)
2025-02-18 15:23:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934084410 ms)
2025-02-18 15:23:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:23:25.400196+00:00', 'runtime': '934080.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.701', 'adc_vl_f': '14.606', 'adc_ic_f': '1.954', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.2', 'ahc_t': '2665.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.687', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934084411 ms)
2025-02-18 15:23:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:22:25.400214+00:00', 'runtime': '934020.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.725', 'adc_vl_f': '14.606', 'adc_ic_f': '1.945', 'adc_il_f': '0.324', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.1', 'ahc_t': '2665.1', 'kwhc': '34.5', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '29.186', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.7', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934084411 ms)
2025-02-18 15:23:35.472 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934093291 ms)
2025-02-18 15:23:35.473 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934093292 ms)
2025-02-18 15:23:35.474 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:23:25.400398+00:00'} (T+934093293 ms)
2025-02-18 15:24:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934144405 ms)
2025-02-18 15:24:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934144406 ms)
2025-02-18 15:24:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:24:25.400170+00:00', 'runtime': '934140.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.692', 'adc_vl_f': '14.606', 'adc_ic_f': '1.945', 'adc_il_f': '0.348', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.2', 'ahc_t': '2665.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.521', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934144407 ms)
2025-02-18 15:24:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:23:25.400196+00:00', 'runtime': '934080.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.701', 'adc_vl_f': '14.606', 'adc_ic_f': '1.954', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.2', 'ahc_t': '2665.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.687', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934144407 ms)
2025-02-18 15:25:26.581 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934204400 ms)
2025-02-18 15:25:26.581 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934204401 ms)
2025-02-18 15:25:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:25:25.400183+00:00', 'runtime': '934200.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.674', 'adc_vl_f': '14.606', 'adc_ic_f': '1.918', 'adc_il_f': '0.355', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.2', 'ahc_t': '2665.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.993', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934204402 ms)
2025-02-18 15:25:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:24:25.400170+00:00', 'runtime': '934140.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.692', 'adc_vl_f': '14.606', 'adc_ic_f': '1.945', 'adc_il_f': '0.348', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.2', 'ahc_t': '2665.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '28.521', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934204402 ms)
2025-02-18 15:26:06.125 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934243944 ms)
2025-02-18 15:26:06.125 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934243945 ms)
2025-02-18 15:26:06.126 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49116)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934243945 ms)
2025-02-18 15:26:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934264414 ms)
2025-02-18 15:26:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934264414 ms)
2025-02-18 15:26:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:26:25.400192+00:00', 'runtime': '934260.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.667', 'adc_vl_f': '14.606', 'adc_ic_f': '1.923', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.3', 'ahc_t': '2665.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.736', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934264415 ms)
2025-02-18 15:26:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:25:25.400183+00:00', 'runtime': '934200.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.674', 'adc_vl_f': '14.606', 'adc_ic_f': '1.918', 'adc_il_f': '0.355', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.2', 'ahc_t': '2665.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.993', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.8', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934264416 ms)
2025-02-18 15:27:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934324411 ms)
2025-02-18 15:27:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934324411 ms)
2025-02-18 15:27:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:27:25.400192+00:00', 'runtime': '934320.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.677', 'adc_vl_f': '14.606', 'adc_ic_f': '1.899', 'adc_il_f': '0.353', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.3', 'ahc_t': '2665.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.872', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934324413 ms)
2025-02-18 15:27:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:26:25.400192+00:00', 'runtime': '934260.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.667', 'adc_vl_f': '14.606', 'adc_ic_f': '1.923', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.3', 'ahc_t': '2665.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.736', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934324413 ms)
2025-02-18 15:27:35.674 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934333493 ms)
2025-02-18 15:27:35.674 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934333494 ms)
2025-02-18 15:27:35.675 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:27:25.400339+00:00'} (T+934333495 ms)
2025-02-18 15:28:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934384406 ms)
2025-02-18 15:28:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934384407 ms)
2025-02-18 15:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:28:25.400219+00:00', 'runtime': '934380.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.655', 'adc_vl_f': '14.606', 'adc_ic_f': '1.884', 'adc_il_f': '0.355', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.3', 'ahc_t': '2665.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.917', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934384408 ms)
2025-02-18 15:28:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:27:25.400192+00:00', 'runtime': '934320.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.677', 'adc_vl_f': '14.606', 'adc_ic_f': '1.899', 'adc_il_f': '0.353', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.3', 'ahc_t': '2665.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.872', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934384408 ms)
2025-02-18 15:28:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934393425 ms)
2025-02-18 15:28:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934393426 ms)
2025-02-18 15:28:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:28:25.400210+00:00'} (T+934393434 ms)
2025-02-18 15:29:16.426 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934434245 ms)
2025-02-18 15:29:16.426 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934434245 ms)
2025-02-18 15:29:16.426 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49318)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934434246 ms)
2025-02-18 15:29:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934444405 ms)
2025-02-18 15:29:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934444405 ms)
2025-02-18 15:29:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:29:25.400190+00:00', 'runtime': '934440.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.674', 'adc_vl_f': '14.606', 'adc_ic_f': '1.802', 'adc_il_f': '0.309', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.555', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934444406 ms)
2025-02-18 15:29:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:28:25.400219+00:00', 'runtime': '934380.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.655', 'adc_vl_f': '14.606', 'adc_ic_f': '1.884', 'adc_il_f': '0.355', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.3', 'ahc_t': '2665.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.917', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '4.9', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934444407 ms)
2025-02-18 15:29:35.584 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934453403 ms)
2025-02-18 15:29:35.584 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934453404 ms)
2025-02-18 15:29:35.585 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:29:25.400178+00:00'} (T+934453405 ms)
2025-02-18 15:30:13.583 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934491402 ms)
2025-02-18 15:30:13.583 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934491402 ms)
2025-02-18 15:30:13.583 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49380)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934491403 ms)
2025-02-18 15:30:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934504417 ms)
2025-02-18 15:30:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934504418 ms)
2025-02-18 15:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:30:25.400277+00:00', 'runtime': '934500.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.664', 'adc_vl_f': '14.606', 'adc_ic_f': '1.809', 'adc_il_f': '0.324', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.238', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934504419 ms)
2025-02-18 15:30:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:29:25.400190+00:00', 'runtime': '934440.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.674', 'adc_vl_f': '14.606', 'adc_ic_f': '1.802', 'adc_il_f': '0.309', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.555', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934504419 ms)
2025-02-18 15:30:42.713 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934520533 ms)
2025-02-18 15:30:42.714 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934520533 ms)
2025-02-18 15:30:42.714 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49412)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934520533 ms)
2025-02-18 15:31:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934564415 ms)
2025-02-18 15:31:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934564416 ms)
2025-02-18 15:31:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:31:25.400181+00:00', 'runtime': '934560.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.732', 'adc_vl_f': '14.606', 'adc_ic_f': '1.819', 'adc_il_f': '0.353', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.981', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934564417 ms)
2025-02-18 15:31:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:30:25.400277+00:00', 'runtime': '934500.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.664', 'adc_vl_f': '14.606', 'adc_ic_f': '1.809', 'adc_il_f': '0.324', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '27.238', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934564418 ms)
2025-02-18 15:32:16.829 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934614648 ms)
2025-02-18 15:32:16.829 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934614648 ms)
2025-02-18 15:32:16.830 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934614649 ms)
2025-02-18 15:32:22.837 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934620657 ms)
2025-02-18 15:32:22.838 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934620657 ms)
2025-02-18 15:32:22.838 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934620657 ms)
2025-02-18 15:32:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934624412 ms)
2025-02-18 15:32:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934624413 ms)
2025-02-18 15:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:32:25.400186+00:00', 'runtime': '934620.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.796', 'adc_vl_f': '14.606', 'adc_ic_f': '1.783', 'adc_il_f': '0.321', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.891', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934624414 ms)
2025-02-18 15:32:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:31:25.400181+00:00', 'runtime': '934560.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.732', 'adc_vl_f': '14.606', 'adc_ic_f': '1.819', 'adc_il_f': '0.353', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.5', 'ahl_t': '1599.5', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.981', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.2', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934624414 ms)
2025-02-18 15:32:35.597 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934633416 ms)
2025-02-18 15:32:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934633417 ms)
2025-02-18 15:32:35.598 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:32:25.400341+00:00'} (T+934633418 ms)
2025-02-18 15:33:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934684410 ms)
2025-02-18 15:33:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934684410 ms)
2025-02-18 15:33:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:33:25.400304+00:00', 'runtime': '934680.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.841', 'adc_vl_f': '14.606', 'adc_ic_f': '1.783', 'adc_il_f': '0.336', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.5', 'ahc_t': '2665.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.664', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934684411 ms)
2025-02-18 15:33:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:32:25.400186+00:00', 'runtime': '934620.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.796', 'adc_vl_f': '14.606', 'adc_ic_f': '1.783', 'adc_il_f': '0.321', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.4', 'ahc_t': '2665.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.891', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.0', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934684412 ms)
2025-02-18 15:33:35.603 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934693423 ms)
2025-02-18 15:33:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934693423 ms)
2025-02-18 15:33:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:33:25.400302+00:00'} (T+934693424 ms)
2025-02-18 15:33:48.041 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934705860 ms)
2025-02-18 15:33:48.041 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934705860 ms)
2025-02-18 15:33:48.042 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49608)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934705861 ms)
2025-02-18 15:34:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934744406 ms)
2025-02-18 15:34:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934744406 ms)
2025-02-18 15:34:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:34:25.400178+00:00', 'runtime': '934740.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.875', 'adc_vl_f': '14.606', 'adc_ic_f': '1.805', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.5', 'ahc_t': '2665.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.136', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934744407 ms)
2025-02-18 15:34:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:33:25.400304+00:00', 'runtime': '934680.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.841', 'adc_vl_f': '14.606', 'adc_ic_f': '1.783', 'adc_il_f': '0.336', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.5', 'ahc_t': '2665.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.664', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934744408 ms)
2025-02-18 15:34:33.180 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934750999 ms)
2025-02-18 15:34:33.180 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934750999 ms)
2025-02-18 15:34:33.181 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49656)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934751000 ms)
2025-02-18 15:34:35.613 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934753432 ms)
2025-02-18 15:34:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934753433 ms)
2025-02-18 15:34:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:34:25.400179+00:00'} (T+934753434 ms)
2025-02-18 15:35:05.286 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934783105 ms)
2025-02-18 15:35:05.286 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934783105 ms)
2025-02-18 15:35:05.286 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49694)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934783106 ms)
2025-02-18 15:35:21.305 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934799125 ms)
2025-02-18 15:35:21.306 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934799125 ms)
2025-02-18 15:35:21.306 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934799126 ms)
2025-02-18 15:35:21.307 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934799127 ms)
2025-02-18 15:35:21.307 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934799127 ms)
2025-02-18 15:35:21.308 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934799127 ms)
2025-02-18 15:35:21.308 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934799128 ms)
2025-02-18 15:35:21.309 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934799128 ms)
2025-02-18 15:35:21.310 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934799129 ms)
2025-02-18 15:35:21.311 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934799130 ms)
2025-02-18 15:35:21.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934799130 ms)
2025-02-18 15:35:21.311 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934799130 ms)
2025-02-18 15:35:21.312 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934799131 ms)
2025-02-18 15:35:21.312 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934799131 ms)
2025-02-18 15:35:21.312 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934799132 ms)
2025-02-18 15:35:21.313 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934799132 ms)
2025-02-18 15:35:21.313 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934799132 ms)
2025-02-18 15:35:21.315 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934799134 ms)
2025-02-18 15:35:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934804419 ms)
2025-02-18 15:35:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934804419 ms)
2025-02-18 15:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:35:25.400205+00:00', 'runtime': '934800.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.988', 'adc_vl_f': '14.606', 'adc_ic_f': '1.761', 'adc_il_f': '0.35', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.5', 'ahc_t': '2665.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.498', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934804420 ms)
2025-02-18 15:35:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:34:25.400178+00:00', 'runtime': '934740.001', 'adc_vb_f': '14.63', 'adc_va_f': '21.875', 'adc_vl_f': '14.606', 'adc_ic_f': '1.805', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.5', 'ahc_t': '2665.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.136', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934804421 ms)
2025-02-18 15:36:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934864414 ms)
2025-02-18 15:36:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934864414 ms)
2025-02-18 15:36:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:36:25.400183+00:00', 'runtime': '934860.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.988', 'adc_vl_f': '14.606', 'adc_ic_f': '1.747', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.6', 'ahc_t': '2665.6', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.0', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934864415 ms)
2025-02-18 15:36:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:35:25.400205+00:00', 'runtime': '934800.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.988', 'adc_vl_f': '14.606', 'adc_ic_f': '1.761', 'adc_il_f': '0.35', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.5', 'ahc_t': '2665.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.498', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.1', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934864418 ms)
2025-02-18 15:36:27.398 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934865217 ms)
2025-02-18 15:36:27.398 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934865217 ms)
2025-02-18 15:36:27.398 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934865218 ms)
2025-02-18 15:36:27.402 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+934865221 ms)
2025-02-18 15:36:27.402 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+934865222 ms)
2025-02-18 15:36:27.403 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934865222 ms)
2025-02-18 15:36:46.493 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+934884312 ms)
2025-02-18 15:36:46.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+934884312 ms)
2025-02-18 15:36:46.493 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 49810)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+934884313 ms)
2025-02-18 15:37:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934924411 ms)
2025-02-18 15:37:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934924412 ms)
2025-02-18 15:37:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:37:25.400168+00:00', 'runtime': '934920.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.037', 'adc_vl_f': '14.606', 'adc_ic_f': '1.718', 'adc_il_f': '0.316', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.6', 'ahc_t': '2665.6', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.215', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934924413 ms)
2025-02-18 15:37:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:36:25.400183+00:00', 'runtime': '934860.001', 'adc_vb_f': '14.627', 'adc_va_f': '21.988', 'adc_vl_f': '14.606', 'adc_ic_f': '1.747', 'adc_il_f': '0.358', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.6', 'ahc_t': '2665.6', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '26.0', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934924413 ms)
2025-02-18 15:38:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+934984411 ms)
2025-02-18 15:38:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+934984411 ms)
2025-02-18 15:38:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:38:25.400178+00:00', 'runtime': '934980.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.083', 'adc_vl_f': '14.606', 'adc_ic_f': '1.698', 'adc_il_f': '0.319', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.6', 'ahc_t': '2665.6', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.547', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934984412 ms)
2025-02-18 15:38:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:37:25.400168+00:00', 'runtime': '934920.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.037', 'adc_vl_f': '14.606', 'adc_ic_f': '1.718', 'adc_il_f': '0.316', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.6', 'ahc_t': '2665.6', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.215', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+934984413 ms)
2025-02-18 15:38:35.629 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934993449 ms)
2025-02-18 15:38:35.630 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+934993449 ms)
2025-02-18 15:38:35.631 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:38:25.400335+00:00'} (T+934993450 ms)
2025-02-18 15:39:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935044408 ms)
2025-02-18 15:39:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935044408 ms)
2025-02-18 15:39:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:39:25.400182+00:00', 'runtime': '935040.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.159', 'adc_vl_f': '14.606', 'adc_ic_f': '1.674', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.683', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935044410 ms)
2025-02-18 15:39:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:38:25.400178+00:00', 'runtime': '934980.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.083', 'adc_vl_f': '14.606', 'adc_ic_f': '1.698', 'adc_il_f': '0.319', 't_hs': '13', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.6', 'ahc_t': '2665.6', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.547', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.2', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935044410 ms)
2025-02-18 15:39:35.795 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935053615 ms)
2025-02-18 15:39:35.796 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935053615 ms)
2025-02-18 15:39:35.797 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:39:25.400184+00:00'} (T+935053616 ms)
2025-02-18 15:40:26.601 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935104420 ms)
2025-02-18 15:40:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935104421 ms)
2025-02-18 15:40:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:40:25.402970+00:00', 'runtime': '935100.004', 'adc_vb_f': '14.627', 'adc_va_f': '22.165', 'adc_vl_f': '14.606', 'adc_ic_f': '1.672', 'adc_il_f': '0.329', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.396', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935104425 ms)
2025-02-18 15:40:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:39:25.400182+00:00', 'runtime': '935040.001', 'adc_vb_f': '14.627', 'adc_va_f': '22.159', 'adc_vl_f': '14.606', 'adc_ic_f': '1.674', 'adc_il_f': '0.321', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.627', 'vb_ref': '14.148', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.683', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935104425 ms)
2025-02-18 15:40:35.608 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935113427 ms)
2025-02-18 15:40:35.608 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935113428 ms)
2025-02-18 15:40:35.609 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:40:25.400345+00:00'} (T+935113429 ms)
2025-02-18 15:41:06.888 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+935144707 ms)
2025-02-18 15:41:06.888 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+935144707 ms)
2025-02-18 15:41:06.889 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50086)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935144708 ms)
2025-02-18 15:41:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935164417 ms)
2025-02-18 15:41:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935164417 ms)
2025-02-18 15:41:26.601 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:41:25.400157+00:00', 'runtime': '935160.001', 'adc_vb_f': '14.63', 'adc_va_f': '22.095', 'adc_vl_f': '14.606', 'adc_ic_f': '1.689', 'adc_il_f': '0.338', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.109', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935164421 ms)
2025-02-18 15:41:26.602 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:40:25.402970+00:00', 'runtime': '935100.004', 'adc_vb_f': '14.627', 'adc_va_f': '22.165', 'adc_vl_f': '14.606', 'adc_ic_f': '1.672', 'adc_il_f': '0.329', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.396', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935164421 ms)
2025-02-18 15:41:52.022 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+935189841 ms)
2025-02-18 15:41:52.022 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+935189841 ms)
2025-02-18 15:41:52.023 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50136)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935189842 ms)
2025-02-18 15:42:26.594 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935224413 ms)
2025-02-18 15:42:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935224414 ms)
2025-02-18 15:42:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:42:25.400721+00:00', 'runtime': '935220.001', 'adc_vb_f': '14.6', 'adc_va_f': '22.006', 'adc_vl_f': '14.606', 'adc_ic_f': '1.689', 'adc_il_f': '0.348', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.621', 'vb_ref': '14.119', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.686', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935224415 ms)
2025-02-18 15:42:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:41:25.400157+00:00', 'runtime': '935160.001', 'adc_vb_f': '14.63', 'adc_va_f': '22.095', 'adc_vl_f': '14.606', 'adc_ic_f': '1.689', 'adc_il_f': '0.338', 't_hs': '14', 't_batt': '14', 't_amb': '14', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.63', 'vb_ref': '14.148', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '25.109', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935224415 ms)
2025-02-18 15:43:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935284412 ms)
2025-02-18 15:43:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935284412 ms)
2025-02-18 15:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:43:25.400254+00:00', 'runtime': '935280.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.988', 'adc_vl_f': '14.603', 'adc_ic_f': '1.686', 'adc_il_f': '0.355', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.603', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.916', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935284413 ms)
2025-02-18 15:43:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:42:25.400721+00:00', 'runtime': '935220.001', 'adc_vb_f': '14.6', 'adc_va_f': '22.006', 'adc_vl_f': '14.606', 'adc_ic_f': '1.689', 'adc_il_f': '0.348', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.621', 'vb_ref': '14.119', 'ahc_r': '2665.7', 'ahc_t': '2665.7', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.686', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.3', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935284414 ms)
2025-02-18 15:44:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935344406 ms)
2025-02-18 15:44:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935344407 ms)
2025-02-18 15:44:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:44:25.402798+00:00', 'runtime': '935340.003', 'adc_vb_f': '14.6', 'adc_va_f': '22.058', 'adc_vl_f': '14.606', 'adc_ic_f': '1.672', 'adc_il_f': '0.353', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.475', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935344408 ms)
2025-02-18 15:44:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:43:25.400254+00:00', 'runtime': '935280.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.988', 'adc_vl_f': '14.603', 'adc_ic_f': '1.686', 'adc_il_f': '0.355', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.603', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.916', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935344408 ms)
2025-02-18 15:44:35.606 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935353425 ms)
2025-02-18 15:44:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935353425 ms)
2025-02-18 15:44:35.607 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:44:25.400165+00:00'} (T+935353426 ms)
2025-02-18 15:44:48.244 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+935366063 ms)
2025-02-18 15:44:48.245 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+935366064 ms)
2025-02-18 15:44:48.245 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935366064 ms)
2025-02-18 15:44:48.246 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+935366065 ms)
2025-02-18 15:44:48.246 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+935366065 ms)
2025-02-18 15:44:48.246 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935366066 ms)
2025-02-18 15:45:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935404403 ms)
2025-02-18 15:45:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935404403 ms)
2025-02-18 15:45:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:45:25.400189+00:00', 'runtime': '935400.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.073', 'adc_vl_f': '14.597', 'adc_ic_f': '1.648', 'adc_il_f': '0.348', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.49', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935404404 ms)
2025-02-18 15:45:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:44:25.402798+00:00', 'runtime': '935340.003', 'adc_vb_f': '14.6', 'adc_va_f': '22.058', 'adc_vl_f': '14.606', 'adc_ic_f': '1.672', 'adc_il_f': '0.353', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.475', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935404404 ms)
2025-02-18 15:45:35.644 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935413463 ms)
2025-02-18 15:45:35.644 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935413463 ms)
2025-02-18 15:45:35.645 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:45:25.400202+00:00'} (T+935413464 ms)
2025-02-18 15:45:52.325 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 101] Network is unreachable (T+935430145 ms)
2025-02-18 15:45:52.326 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 101] Network is unreachable (T+935430145 ms)
2025-02-18 15:45:52.327 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 0)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935430146 ms)
2025-02-18 15:46:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935464417 ms)
2025-02-18 15:46:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935464418 ms)
2025-02-18 15:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:46:25.400172+00:00', 'runtime': '935460.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.198', 'adc_vl_f': '14.59', 'adc_ic_f': '1.611', 'adc_il_f': '0.333', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.505', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935464419 ms)
2025-02-18 15:46:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:45:25.400189+00:00', 'runtime': '935400.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.073', 'adc_vl_f': '14.597', 'adc_ic_f': '1.648', 'adc_il_f': '0.348', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.49', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935464419 ms)
2025-02-18 15:46:35.614 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935473433 ms)
2025-02-18 15:46:35.614 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935473434 ms)
2025-02-18 15:46:35.615 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:46:25.400154+00:00'} (T+935473435 ms)
2025-02-18 15:46:54.457 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+935492276 ms)
2025-02-18 15:46:54.457 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+935492276 ms)
2025-02-18 15:46:54.458 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50466)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935492277 ms)
2025-02-18 15:46:57.536 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+935495355 ms)
2025-02-18 15:46:57.536 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+935495356 ms)
2025-02-18 15:46:57.538 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50470)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935495357 ms)
2025-02-18 15:47:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935524416 ms)
2025-02-18 15:47:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935524417 ms)
2025-02-18 15:47:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:47:25.402770+00:00', 'runtime': '935520.003', 'adc_vb_f': '14.597', 'adc_va_f': '22.095', 'adc_vl_f': '14.6', 'adc_ic_f': '1.582', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.9', 'ahc_t': '2665.9', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.007', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935524418 ms)
2025-02-18 15:47:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:46:25.400172+00:00', 'runtime': '935460.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.198', 'adc_vl_f': '14.59', 'adc_ic_f': '1.611', 'adc_il_f': '0.333', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.8', 'ahc_t': '2665.8', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.505', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.4', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935524418 ms)
2025-02-18 15:47:35.728 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935533548 ms)
2025-02-18 15:47:35.729 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935533548 ms)
2025-02-18 15:47:35.730 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:47:25.400188+00:00'} (T+935533549 ms)
2025-02-18 15:48:26.592 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935584411 ms)
2025-02-18 15:48:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935584411 ms)
2025-02-18 15:48:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:48:25.402696+00:00', 'runtime': '935580.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.771', 'adc_vl_f': '14.606', 'adc_ic_f': '1.59', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.9', 'ahc_t': '2665.9', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.72', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935584416 ms)
2025-02-18 15:48:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:47:25.402770+00:00', 'runtime': '935520.003', 'adc_vb_f': '14.597', 'adc_va_f': '22.095', 'adc_vl_f': '14.6', 'adc_ic_f': '1.582', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.9', 'ahc_t': '2665.9', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '24.007', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935584416 ms)
2025-02-18 15:48:35.595 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935593415 ms)
2025-02-18 15:48:35.596 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935593415 ms)
2025-02-18 15:48:35.597 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:48:25.400084+00:00'} (T+935593416 ms)
2025-02-18 15:49:26.605 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935644425 ms)
2025-02-18 15:49:26.606 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935644425 ms)
2025-02-18 15:49:26.610 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:49:25.400178+00:00', 'runtime': '935640.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.643', 'adc_vl_f': '14.603', 'adc_ic_f': '1.606', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2665.9', 'ahc_t': '2665.9', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.72', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935644430 ms)
2025-02-18 15:49:26.611 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:48:25.402696+00:00', 'runtime': '935580.003', 'adc_vb_f': '14.6', 'adc_va_f': '21.771', 'adc_vl_f': '14.606', 'adc_ic_f': '1.59', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2665.9', 'ahc_t': '2665.9', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.72', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935644430 ms)
2025-02-18 15:49:35.602 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935653421 ms)
2025-02-18 15:49:35.603 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935653422 ms)
2025-02-18 15:49:35.604 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:49:25.405131+00:00'} (T+935653423 ms)
2025-02-18 15:50:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935704405 ms)
2025-02-18 15:50:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935704405 ms)
2025-02-18 15:50:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:50:25.400139+00:00', 'runtime': '935700.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.594', 'adc_ic_f': '1.573', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.614', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935704409 ms)
2025-02-18 15:50:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:49:25.400178+00:00', 'runtime': '935640.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.643', 'adc_vl_f': '14.603', 'adc_ic_f': '1.606', 'adc_il_f': '0.336', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2665.9', 'ahc_t': '2665.9', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.72', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.5', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935704410 ms)
2025-02-18 15:50:35.585 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935713405 ms)
2025-02-18 15:50:35.586 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935713405 ms)
2025-02-18 15:50:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:50:25.400185+00:00'} (T+935713410 ms)
2025-02-18 15:51:26.599 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935764419 ms)
2025-02-18 15:51:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935764419 ms)
2025-02-18 15:51:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:51:25.400189+00:00', 'runtime': '935760.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.664', 'adc_vl_f': '14.603', 'adc_ic_f': '1.59', 'adc_il_f': '0.338', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.554', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935764423 ms)
2025-02-18 15:51:26.604 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:50:25.400139+00:00', 'runtime': '935700.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.558', 'adc_vl_f': '14.594', 'adc_ic_f': '1.573', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.6', 'ahl_t': '1599.6', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.614', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.3', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935764423 ms)
2025-02-18 15:51:35.582 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935773401 ms)
2025-02-18 15:51:35.582 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935773402 ms)
2025-02-18 15:51:35.584 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:51:25.400212+00:00'} (T+935773403 ms)
2025-02-18 15:52:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935824415 ms)
2025-02-18 15:52:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935824415 ms)
2025-02-18 15:52:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:52:25.400434+00:00', 'runtime': '935820.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.774', 'adc_vl_f': '14.6', 'adc_ic_f': '1.606', 'adc_il_f': '0.367', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.342', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935824416 ms)
2025-02-18 15:52:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:51:25.400189+00:00', 'runtime': '935760.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.664', 'adc_vl_f': '14.603', 'adc_ic_f': '1.59', 'adc_il_f': '0.338', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.554', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935824417 ms)
2025-02-18 15:52:30.427 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
 (T+935828247 ms)
2025-02-18 15:52:37.047 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+935834867 ms)
2025-02-18 15:52:37.048 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+935834867 ms)
2025-02-18 15:52:37.048 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50830)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935834867 ms)
2025-02-18 15:53:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935884410 ms)
2025-02-18 15:53:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935884411 ms)
2025-02-18 15:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:53:25.400253+00:00', 'runtime': '935880.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.884', 'adc_vl_f': '14.603', 'adc_ic_f': '1.585', 'adc_il_f': '0.353', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.237', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935884412 ms)
2025-02-18 15:53:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:52:25.400434+00:00', 'runtime': '935820.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.774', 'adc_vl_f': '14.6', 'adc_ic_f': '1.606', 'adc_il_f': '0.367', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.342', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935884412 ms)
2025-02-18 15:54:02.254 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+935920073 ms)
2025-02-18 15:54:02.254 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+935920074 ms)
2025-02-18 15:54:02.255 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 50920)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+935920074 ms)
2025-02-18 15:54:26.588 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+935944407 ms)
2025-02-18 15:54:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+935944407 ms)
2025-02-18 15:54:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:54:25.400186+00:00', 'runtime': '935940.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.765', 'adc_vl_f': '14.6', 'adc_ic_f': '1.561', 'adc_il_f': '0.348', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.131', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935944408 ms)
2025-02-18 15:54:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:53:25.400253+00:00', 'runtime': '935880.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.884', 'adc_vl_f': '14.603', 'adc_ic_f': '1.585', 'adc_il_f': '0.353', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.0', 'ahc_t': '2666.0', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20146', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.237', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.6', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+935944409 ms)
2025-02-18 15:54:35.601 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935953420 ms)
2025-02-18 15:54:35.601 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+935953421 ms)
2025-02-18 15:54:35.602 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:54:25.400336+00:00'} (T+935953422 ms)
2025-02-18 15:55:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936004405 ms)
2025-02-18 15:55:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936004405 ms)
2025-02-18 15:55:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:55:25.400197+00:00', 'runtime': '936000.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.649', 'adc_vl_f': '14.6', 'adc_ic_f': '1.544', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.025', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936004406 ms)
2025-02-18 15:55:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:54:25.400186+00:00', 'runtime': '935940.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.765', 'adc_vl_f': '14.6', 'adc_ic_f': '1.561', 'adc_il_f': '0.348', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.131', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936004407 ms)
2025-02-18 15:55:35.590 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936013409 ms)
2025-02-18 15:55:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936013410 ms)
2025-02-18 15:55:35.595 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:55:25.400216+00:00'} (T+936013415 ms)
2025-02-18 15:56:26.518 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936064338 ms)
2025-02-18 15:56:26.519 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936064338 ms)
2025-02-18 15:56:26.519 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51084)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936064338 ms)
2025-02-18 15:56:26.598 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936064417 ms)
2025-02-18 15:56:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936064418 ms)
2025-02-18 15:56:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:56:25.400253+00:00', 'runtime': '936060.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.594', 'adc_ic_f': '1.541', 'adc_il_f': '0.326', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.237', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936064419 ms)
2025-02-18 15:56:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:55:25.400197+00:00', 'runtime': '936000.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.649', 'adc_vl_f': '14.6', 'adc_ic_f': '1.544', 'adc_il_f': '0.331', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.025', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936064419 ms)
2025-02-18 15:56:34.570 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936072389 ms)
2025-02-18 15:56:34.570 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936072390 ms)
2025-02-18 15:56:34.571 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51096)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936072390 ms)
2025-02-18 15:56:35.589 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936073408 ms)
2025-02-18 15:56:35.589 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936073409 ms)
2025-02-18 15:56:35.594 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:56:25.400251+00:00'} (T+936073413 ms)
2025-02-18 15:57:26.597 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936124417 ms)
2025-02-18 15:57:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936124417 ms)
2025-02-18 15:57:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:57:25.400175+00:00', 'runtime': '936120.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.481', 'adc_vl_f': '14.587', 'adc_ic_f': '1.517', 'adc_il_f': '0.324', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.663', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936124418 ms)
2025-02-18 15:57:26.600 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:56:25.400253+00:00', 'runtime': '936060.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.558', 'adc_vl_f': '14.594', 'adc_ic_f': '1.541', 'adc_il_f': '0.326', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '23.237', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936124419 ms)
2025-02-18 15:57:35.598 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936133417 ms)
2025-02-18 15:57:35.599 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936133418 ms)
2025-02-18 15:57:35.600 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:57:25.400214+00:00'} (T+936133419 ms)
2025-02-18 15:58:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936184412 ms)
2025-02-18 15:58:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936184413 ms)
2025-02-18 15:58:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:58:25.400178+00:00', 'runtime': '936180.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.423', 'adc_vl_f': '14.578', 'adc_ic_f': '1.505', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.678', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936184414 ms)
2025-02-18 15:58:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:57:25.400175+00:00', 'runtime': '936120.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.481', 'adc_vl_f': '14.587', 'adc_ic_f': '1.517', 'adc_il_f': '0.324', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.1', 'ahc_t': '2666.1', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.663', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.7', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936184414 ms)
2025-02-18 15:58:35.589 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936193408 ms)
2025-02-18 15:58:35.590 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936193409 ms)
2025-02-18 15:58:35.591 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:58:25.400214+00:00'} (T+936193410 ms)
2025-02-18 15:59:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936244409 ms)
2025-02-18 15:59:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936244410 ms)
2025-02-18 15:59:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 15:59:25.400551+00:00', 'runtime': '936240.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.46', 'adc_vl_f': '14.578', 'adc_ic_f': '1.493', 'adc_il_f': '0.333', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.497', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936244411 ms)
2025-02-18 15:59:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:58:25.400178+00:00', 'runtime': '936180.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.423', 'adc_vl_f': '14.578', 'adc_ic_f': '1.505', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.678', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936244411 ms)
2025-02-18 15:59:35.461 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936253280 ms)
2025-02-18 15:59:35.461 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936253281 ms)
2025-02-18 15:59:35.462 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 15:59:25.400236+00:00'} (T+936253282 ms)
2025-02-18 16:00:24.933 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936302752 ms)
2025-02-18 16:00:24.933 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936302752 ms)
2025-02-18 16:00:24.933 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51342)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936302753 ms)
2025-02-18 16:00:26.587 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936304406 ms)
2025-02-18 16:00:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936304407 ms)
2025-02-18 16:00:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:00:25.400149+00:00', 'runtime': '936300.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.512', 'adc_vl_f': '14.59', 'adc_ic_f': '1.5', 'adc_il_f': '0.324', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.391', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936304408 ms)
2025-02-18 16:00:26.590 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 15:59:25.400551+00:00', 'runtime': '936240.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.46', 'adc_vl_f': '14.578', 'adc_ic_f': '1.493', 'adc_il_f': '0.333', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.497', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936304409 ms)
2025-02-18 16:00:33.477 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Rsync failure: ssh: Could not resolve hostname hamma.dev: Temporary failure in name resolution
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
 (T+936311296 ms)
2025-02-18 16:00:52.069 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936329888 ms)
2025-02-18 16:00:52.069 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936329888 ms)
2025-02-18 16:00:52.069 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51372)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936329889 ms)
2025-02-18 16:01:05.206 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936343025 ms)
2025-02-18 16:01:05.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936343025 ms)
2025-02-18 16:01:05.206 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51386)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936343026 ms)
2025-02-18 16:01:26.585 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936364404 ms)
2025-02-18 16:01:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936364404 ms)
2025-02-18 16:01:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:01:25.400192+00:00', 'runtime': '936360.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.536', 'adc_vl_f': '14.584', 'adc_ic_f': '1.498', 'adc_il_f': '0.343', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.195', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936364406 ms)
2025-02-18 16:01:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:00:25.400149+00:00', 'runtime': '936300.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.512', 'adc_vl_f': '14.59', 'adc_ic_f': '1.5', 'adc_il_f': '0.324', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.391', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936364407 ms)
2025-02-18 16:01:30.347 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936368167 ms)
2025-02-18 16:01:30.348 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936368167 ms)
2025-02-18 16:01:30.348 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51412)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936368167 ms)
2025-02-18 16:02:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936424414 ms)
2025-02-18 16:02:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936424415 ms)
2025-02-18 16:02:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:02:25.400577+00:00', 'runtime': '936420.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '1.449', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.135', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936424416 ms)
2025-02-18 16:02:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:01:25.400192+00:00', 'runtime': '936360.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.536', 'adc_vl_f': '14.584', 'adc_ic_f': '1.498', 'adc_il_f': '0.343', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.2', 'ahc_t': '2666.2', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.195', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.8', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936424419 ms)
2025-02-18 16:02:44.530 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936442350 ms)
2025-02-18 16:02:44.531 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936442350 ms)
2025-02-18 16:02:44.531 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51490)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936442350 ms)
2025-02-18 16:03:26.595 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936484414 ms)
2025-02-18 16:03:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936484414 ms)
2025-02-18 16:03:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:03:25.400174+00:00', 'runtime': '936480.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.777', 'adc_vl_f': '14.6', 'adc_ic_f': '1.442', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.029', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936484415 ms)
2025-02-18 16:03:26.599 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:02:25.400577+00:00', 'runtime': '936420.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.667', 'adc_vl_f': '14.603', 'adc_ic_f': '1.449', 'adc_il_f': '0.292', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.135', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936484418 ms)
2025-02-18 16:03:28.656 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936486475 ms)
2025-02-18 16:03:28.656 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936486475 ms)
2025-02-18 16:03:28.657 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51538)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936486476 ms)
2025-02-18 16:03:35.604 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936493424 ms)
2025-02-18 16:03:35.605 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Error details:
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 90, in execute
    out = self.rsync()
  File "/home/pi/dev/mjolnir-hamma/plugins/rsync_server.py", line 141, in rsync
    timeout=TIMEOUT,
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+936493424 ms)
2025-02-18 16:03:35.606 | INFO | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | Current data: {'time': '2025-02-18 16:03:25.405067+00:00'} (T+936493426 ms)
2025-02-18 16:04:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936544411 ms)
2025-02-18 16:04:26.592 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936544411 ms)
2025-02-18 16:04:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:04:25.400188+00:00', 'runtime': '936540.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.887', 'adc_vl_f': '14.603', 'adc_ic_f': '1.464', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.104', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936544416 ms)
2025-02-18 16:04:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:03:25.400174+00:00', 'runtime': '936480.001', 'adc_vb_f': '14.597', 'adc_va_f': '21.777', 'adc_vl_f': '14.6', 'adc_ic_f': '1.442', 'adc_il_f': '0.316', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.029', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936544416 ms)
2025-02-18 16:05:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936604408 ms)
2025-02-18 16:05:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936604408 ms)
2025-02-18 16:05:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:05:25.400783+00:00', 'runtime': '936600.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.991', 'adc_vl_f': '14.6', 'adc_ic_f': '1.483', 'adc_il_f': '0.353', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.636', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936604412 ms)
2025-02-18 16:05:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:04:25.400188+00:00', 'runtime': '936540.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.887', 'adc_vl_f': '14.603', 'adc_ic_f': '1.464', 'adc_il_f': '0.319', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '22.104', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936604413 ms)
2025-02-18 16:06:15.974 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936653793 ms)
2025-02-18 16:06:15.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936653793 ms)
2025-02-18 16:06:15.974 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51726)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936653794 ms)
2025-02-18 16:06:26.586 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936664405 ms)
2025-02-18 16:06:26.586 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936664406 ms)
2025-02-18 16:06:26.587 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:06:25.400626+00:00', 'runtime': '936660.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.128', 'adc_vl_f': '14.594', 'adc_ic_f': '1.493', 'adc_il_f': '0.36', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.349', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936664407 ms)
2025-02-18 16:06:26.588 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:05:25.400783+00:00', 'runtime': '936600.001', 'adc_vb_f': '14.6', 'adc_va_f': '21.991', 'adc_vl_f': '14.6', 'adc_ic_f': '1.483', 'adc_il_f': '0.353', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.3', 'ahc_t': '2666.3', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.636', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '5.9', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936664407 ms)
2025-02-18 16:06:51.078 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936688897 ms)
2025-02-18 16:06:51.078 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936688897 ms)
2025-02-18 16:06:51.078 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51764)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936688897 ms)
2025-02-18 16:07:26.583 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936724402 ms)
2025-02-18 16:07:26.583 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936724403 ms)
2025-02-18 16:07:26.584 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:07:25.400183+00:00', 'runtime': '936720.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.269', 'adc_vl_f': '14.584', 'adc_ic_f': '1.478', 'adc_il_f': '0.358', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.742', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936724404 ms)
2025-02-18 16:07:26.585 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:06:25.400626+00:00', 'runtime': '936660.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.128', 'adc_vl_f': '14.594', 'adc_ic_f': '1.493', 'adc_il_f': '0.36', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.349', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936724404 ms)
2025-02-18 16:08:26.596 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936784416 ms)
2025-02-18 16:08:26.597 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936784416 ms)
2025-02-18 16:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:08:25.400194+00:00', 'runtime': '936780.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.305', 'adc_vl_f': '14.581', 'adc_ic_f': '1.476', 'adc_il_f': '0.355', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.425', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936784417 ms)
2025-02-18 16:08:26.598 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:07:25.400183+00:00', 'runtime': '936720.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.269', 'adc_vl_f': '14.584', 'adc_ic_f': '1.478', 'adc_il_f': '0.358', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.742', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936784418 ms)
2025-02-18 16:09:26.593 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936844412 ms)
2025-02-18 16:09:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936844412 ms)
2025-02-18 16:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:09:25.400188+00:00', 'runtime': '936840.001', 'adc_vb_f': '14.6', 'adc_va_f': '22.205', 'adc_vl_f': '14.594', 'adc_ic_f': '1.462', 'adc_il_f': '0.35', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.455', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936844413 ms)
2025-02-18 16:09:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:08:25.400194+00:00', 'runtime': '936780.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.305', 'adc_vl_f': '14.581', 'adc_ic_f': '1.476', 'adc_il_f': '0.355', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.425', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936844414 ms)
2025-02-18 16:09:43.367 | ERROR | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | OSError with socket: [Errno 113] No route to host (T+936861186 ms)
2025-02-18 16:09:43.367 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/brokkr/src/brokkr/utils/network.py", line 111, in setup_socket
    getattr(sock, action)(address_tuple)
OSError: [Errno 113] No route to host (T+936861187 ms)
2025-02-18 16:09:43.368 | INFO | HAMMA2 AGS Science Data Ingest Process | brokkr.utils.network | Socket details: {'socket': <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('48.187.7.218', 51946)>, 'address': ('10.10.10.1', 8080), 'action': 'connect'} (T+936861187 ms)
2025-02-18 16:10:26.591 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936904410 ms)
2025-02-18 16:10:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936904411 ms)
2025-02-18 16:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:10:25.400163+00:00', 'runtime': '936900.001', 'adc_vb_f': '14.6', 'adc_va_f': '22.198', 'adc_vl_f': '14.594', 'adc_ic_f': '1.413', 'adc_il_f': '0.309', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.5', 'ahc_t': '2666.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.183', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.1', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936904415 ms)
2025-02-18 16:10:26.596 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:09:25.400188+00:00', 'runtime': '936840.001', 'adc_vb_f': '14.6', 'adc_va_f': '22.205', 'adc_vl_f': '14.594', 'adc_ic_f': '1.462', 'adc_il_f': '0.35', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.4', 'ahc_t': '2666.4', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.455', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.0', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936904415 ms)
2025-02-18 16:11:26.590 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+936964410 ms)
2025-02-18 16:11:26.591 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+936964410 ms)
2025-02-18 16:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:11:25.400164+00:00', 'runtime': '936960.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.211', 'adc_vl_f': '14.587', 'adc_ic_f': '1.389', 'adc_il_f': '0.312', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.5', 'ahc_t': '2666.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.8', 'ahl_t': '1599.8', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.912', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.1', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936964414 ms)
2025-02-18 16:11:26.595 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:10:25.400163+00:00', 'runtime': '936900.001', 'adc_vb_f': '14.6', 'adc_va_f': '22.198', 'adc_vl_f': '14.594', 'adc_ic_f': '1.413', 'adc_il_f': '0.309', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.597', 'vb_ref': '14.119', 'ahc_r': '2666.5', 'ahc_t': '2666.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.7', 'ahl_t': '1599.7', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.183', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.1', 'ahl_daily': '6.4', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+936964415 ms)
2025-02-18 16:12:26.589 | ERROR | Telemetry Process | plugins.state_monitor.StateMonitor | TypeError evaluating in <class 'plugins.state_monitor.StateMonitor'> on step State Monitor: '<' not supported between instances of 'str' and 'int' (T+937024408 ms)
2025-02-18 16:12:26.589 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Error details:
Traceback (most recent call last):
  File "/home/pi/dev/mjolnir-hamma/plugins/state_monitor.py", line 124, in execute
    if (space_now < self.low_space) and (space_pre > self.low_space):
TypeError: '<' not supported between instances of 'str' and 'int' (T+937024408 ms)
2025-02-18 16:12:26.593 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Current data: {'time': '2025-02-18 16:12:25.402776+00:00', 'runtime': '937020.003', 'adc_vb_f': '14.597', 'adc_va_f': '22.095', 'adc_vl_f': '14.597', 'adc_ic_f': '1.399', 'adc_il_f': '0.314', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.5', 'ahc_t': '2666.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.8', 'ahl_t': '1599.8', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '21.304', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.1', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+937024413 ms)
2025-02-18 16:12:26.594 | INFO | Telemetry Process | plugins.state_monitor.StateMonitor | Previous data: {'time': '2025-02-18 16:11:25.400164+00:00', 'runtime': '936960.001', 'adc_vb_f': '14.597', 'adc_va_f': '22.211', 'adc_vl_f': '14.587', 'adc_ic_f': '1.389', 'adc_il_f': '0.312', 't_hs': '14', 't_batt': '15', 't_amb': '15', 't_rts': '25', 'charge_state': '6', 'array_fault': '0', 'vb_f': '14.6', 'vb_ref': '14.119', 'ahc_r': '2666.5', 'ahc_t': '2666.5', 'kwhc': '34.6', 'load_state': '1', 'load_fault': '0', 'v_lvd': '11.499', 'ahl_r': '1599.8', 'ahl_t': '1599.8', 'hourmeter': '20147', 'alarm': '1', 'dip_switch': '13', 'led_state': '5', 'power_out': '20.912', 'sweep_vmp': '19.885', 'sweep_pmax': '41.385', 'sweep_voc': '22.302', 'vb_min_daily': '12.482', 'vb_max_daily': '14.767', 'ahc_daily': '6.1', 'ahl_daily': '6.5', 'array_fault_daily': '0', 'load_fault_daily': '0', 'alarm_daily': '1', 'vb_min': '5.905', 'vb_max': '20.898', 'ping': '1', 'sequence_count': 'NA', 'timestamp': 'NA', 'crc_errors': 'NA', 'valid_packets': 'NA', 'bytes_read': 'NA', 'bytes_written': 'NA', 'bytes_remaining': 'NA', 'packets_sent': 'NA', 'packets_dropped': 'NA'} (T+937024413 ms)
2025-02-18 16:12:35.618 | ERROR | Rsync HAMMA realtime info to server Process | plugins.rsync_server.RsyncServer | TimeoutExpired evaluating in <class 'plugins.rsync_server.RsyncServer'> on step Rsync HAMMA plot: Command '['rsync', '-avz', '--include=*.png', '--exclude=*', '/home/pi/brokkr/hamma/trigger_plots/', 'pi@hamma.dev:/var/www/hamma.dev/public_html/latest']' timed out after 10 seconds (T+937033437 ms)